How Primus Builders Started Harnessing Data

Challenge

Primus was using DOMO as their BI solution, which gave their team a way to build reports and govern the data available to them. However, there are some data sources that do not have integrations to load their data into a data warehouse.

There were 4 sources that needed integration: Procore, SafetyAmp, Paycom and Oracle Primavera Cloud. Each has a REST API, but Primus needed the data as relational table in a data warehouse that they could connect to the DOMO platform.

The project had the following goals:

Catch Payroll Mistakes

Primus wanted to monitor their payroll punches from Paycom for patterns that just didn’t look right and needed further investigation. By replicating the data into a data warehouse, their analyst can set up notifications that alert HR and project managers when different scenarios crop up, and correct them before the next payroll period.

Evaluate Cost Controls

Primus wanted to be able to use budget line items from Procore to create cost reports and compare them to past projects, as well as other parts of the same project. By learning from what went well, and didn’t go well, there’s opportunity an to assign hard costs to management styles, and course-correct when necessary.

Safety Reporting

Primus wanted to pull SafetyAmp forms that are unique to each job, and group the answers to similar types of questions. That way, they can get alerts when any violations are out of the norm, and identify patterns that can shape safety policy more broadly.

Create a Project Health Scorecard

Using data from Oracle Primavera, and the data above, Primus can move forward to create a unified health scorecard that shows all stakeholders how the project is progressing in real-time.

"Before we started working with Aquifer, there was no application to centrally look to see the life score of a project–to see how we were doing in reference to schedule and cost and quality. There was no way to take all those pieces and understand them holistically."
- John Robba (CTO, Primus Builders)

Solution

Replicate data from each application into a data warehouse

With each of the four integration sources, the type of integration was a REST API. This means that the data Primus was looking for was available as JSON, but not as a relational SQL table, much like a spreadsheet.

Aquifer was able to develop an integration tailored to each of the sources, Procore, SafetyAmp, Paycom, and Primavera. With each source, there are several factors that are unique to the API and the way Aquifer connects.

For example:

• There are differences in how the connection is authorized.
• The data in raw form from the API may be different from the desired model in the destination database.
• The integration must be aware of and mitigate the affects of rate-limiting.
• Some APIs are simply undocumented and data must be retrieved through research, communication with the source’s support team, and trial and error.

Aquifer worked with the Primus team to model the destination data and clarify the data's lineage

After the data from each source was replicating consistently and reports had been created, the Aquifer team was available to support the Primus BI team. Aquifer was able to clarify or resolve any issues related to unexpected data types, edge cases, and issues that may affect the cadence of data replication.

Results

Reduction in manual data loading time

As reports began circulating throughout the organization, it became clear that the BI team at Primus had hit the mark. The team had been able to reduce time dedicated to manually downloading, transforming, and uploading CSVs between sources and the data warehouse. This meant staff could focus on higher-level business requirements while the extraction and loading of data was automated.

Production of previously unavailable reports

This project affected several departments, including Safety, Accounting, Operations, and the Executive team. The members of these departments were each thrilled to consistent, up-to-date information to make better decisions with their teams.

Increased confidence in data engineering projects

The Primus team had proven that data engineering projects could be successfully executed within their organization. And, with that success, departments began to brainstorm new questions they would like to ask of even more data sources.

"Instead of hiring more and more people for rudimentary tasks like filing papers and running spreadsheets, now we have a better approach with data and can display exactly what we need, while also setting up triggers and notifications. Now that time can be utilized for something else. And that's really how you grow."
- John Robba (CTO, Primus Builders)

Future

More sources, more reports, deeper integration

As it became clear within the organization that sources could be added to the BI environment, the requests started to flood in to the BI team. And in addition to sources, deeper and more complex questions continue be asked of the current data set, often combining data from multiple sources into one report.

Primus Builders is on it’s way to developing a broader data engineering strategy, and is looking to leverage more of Aquifer’s integration services in the future.