TL,DR : the report is here. and I appreciate a vote on this bug report
First, don’t be excited, it is a silly workaround, and introduce it is own problem, but if you are like me and need to deliver a nice-looking pivot table in Google Data Studio, it may be worth the hassle.
The Problem.
Show the spent and budget by Campaign and country, the spent is at the country level, the budget at the country level, here is a sample data set.
The Solution, First try
Probably you are saying, it is too easy, why you are writing a blog about it, GDS support pivot Table !!, let’s see the result
We have three Problems already (1 bug, 1 limitation and 1 by design)
Bug: you can return not return a null in the metric spent
By design: GDS does not understand hierarchy, country null is all good.
Limitation: The Famous Excel compact View is not supported
Here is the deal, contrary to what you may read in the internet, Pivot table is the most used viz in reporting ( ok, maybe second after table) and users will want their pivot table to look exactly like their beloved Excel, my own experience, if you show a user a map for example and he ask for a feature which is not possible, you can say, I can’t do it and people will tolerate that, but their Excel looking Pivot table, zero tolerance, if you can’t reproduce it, they will either think :
Your BI is not good
You don’t know the tool
The Solution, SQL!!.
Write a SQL that return a column that show the campaign and country in the same field, using union
Assuming your data is on google sheet.
Link Google Sheet to an External table in BigQuery
2-Write the Query
Connect to that table using a custom Query
SELECT project,sum(budget) as budget,sum(spent) as spent FROM `test-187010.work.factraw`
group by 1
union all
SELECT Concat(“\U0001f680”,country), budget, spent FROM `test-187010.work.factraw` where country IS NOT NULL
3-BI engine does not support External table.
Every time you open the report, GDS studio will issue a new query which cost 10 mb minimum !!!, even if the data is 1 kb ( it is a big data thing after all), to avoid that we extract the data
4-Profit 🙂
We use conditional formatting to highlight the row campaign.
needless to say !!!! you should not use it unless you have to, cross filtering will be a mess , Hopefully GDS will improve pivot table formatting in the near future.
I love Google data Studio, I am using it for this project nemtracker.github.io and it is perfect for this use case.
For other Dashboard at Work, I am using PowerBI, for no reason, I tried a new experiment, can GDS be used instead, it is an academic exercise only. the use case is a typical Business Reports, a lot of different small datasets with different granularity , something like budget vs items sold etc.
I am just recording the pain points in GDS , the good thing, most of them are under development, and some of them overlap, for example if we have Parameters controls, probably, there will be less need for blending ( which is very limited at the moment).
it is not a critique, GDS has some killer features, I particular like custom visuals as there are no limit the number of data plotted which is a pain in other Software.
the assumption is all the data-sources is already loaded and cleaned and ready to be analysed in BigQuery.
TL,DR : the pain points are at the Calculation level , Obviously if all you data is at the same granularity, then everything is easy my conclusion, nearly there !!!, but will revise when Parameter controls are supported.
Instead of Writing a full blog, I thought showing a report is a more practical approach
How is an engineering deliverable list a bit of a misnomer? Find out of a web page with editable fields can streamline the tracking of progress on list items.
I am a firm believer in the use of a standard project website portal from which the project team at large can quickly access key data and metrics about the project. This is not meant to be confused with a project dashboard, or PowerBI visualization. This is simple html file linked to a database with key flat-file information. Using smart JavaScript code, it is also possible to EDIT some of the key information.
With everything, before we even begin, we want to focus on what core digital strategies we are trying to tackle. Again, the project website is just a tactical approach, underlying it are the more relevant strategic goals we want to operate under.
The key data sources involved with engineering progress will be:
Engineering Deliverable List
Manhours per WBS
Your engineering deliverable list is a bit of a misnomer. A lot of engineering and design tasks are not specifically related to a “deliverable.” In my view of the world, while you will have specific deliverables and need to track them, your progress list should also include everything else you are doing too – up to a point where perhaps the level of detail is too small.
For analysis, productivity factors for engineering are critical. As such, this post would not be complete without a discussion and visibility into hours and productivity factors.
The Menu
Our menu follows many of the key functions of project controls. However this specific post will dive into the ENGINEERING section only.
If we expand our menu, we see various views into our data: by project, discipline, contracts, schedule IDs and a unique view for controls engineering
Detail Views / EDIT MODE
Perhaps before we look at our summary and drill downs, the main control screen we would use on a day to day basis would be the Deliverables by WBS and Discipline
In the below screen, we can see all out progress items, our budget hours, progress% as well as the mapping to P6 ID (as all progress items should be mapped to into your schedule)
This view into deliverables by itself is not specifically unique. Where the magic happens is when we enter EDIT MODE.
Inside our EDIT MODE, we can directly update the progress %, and update our mapping to schedule ID. These features in my mind are your killer features that distinguish this from any other app. The shear ease in updating items in this way is a breath of fresh air, not to mention the live feedback and visibility throughout your team.
JIRA
Each combination of WBS and Discipline can be mapped to a JIRA task. Obviously we know our WBS can be considered an EPIC. The power in using a tool such as JIRA is that we can now track more detailed L4/5 tasks using JIRA Subtasks. These are completely flexible to allow the user to manage their own tasks.
Additionally, the ability to embed commentary and status is a brilliant way to distribute and communicate key status and blockers if any.
Summary Pages
Now that we have seen how we will actually interact with a project website, we can now showcase the summary level reporting. Specifics of how you report can often times be better captured using an analytics platform like PowerBI; however, more often then not, simple summaries and metrics are what drive our business.
In the summary by project, we can easily see key metrics per project. Each project would have a link to allow us to drill into the details for each project
In the above, we are now looking at the individual functional delivery areas (disciplines) for our project. Again we can see key budgets, spent hours, progress and productivity measures too. By clicking each discipline, we further dive into our project by now looking into the detailed WBS elements for that Discipline
It is in this view that we can see detailed metrics per WBS/Discipline pair.
Understanding our Spent Hours from time sheets is a critical management function. As such, the SAP actuals here is a link that takes us to a screen where we can view the detailed weekly time sheet data to see who (and when) has book to this element.
Conclusion
In summary we started with some key strategic thinking and built out a tool that ticks a lot of boxes in the EPCM construction world.
More often then not, I am confronted with technology that “shows me” but doesn’t allow me to interact, edit, or collaborate. Everyone wants to solve the “one source of truth” however, information changes and is updated by people. That is the missing link in a lot of our data analytics. Look into the work processes that generate information in the first place. Look at what people need to better capture their raw information into a data format and platform that others can now use.
This approach to put management of engineering items to plain sight, with ease of access is just one approach. There are 100s of ways to approach this, but if you stick to core strategic ideals, you can’t go wrong.
Digital Strategy is an enabler to allow you to tackle problems, however, unless you have a clear vision for the required data integration that follows, it doesn’t make a difference what your strategy is. What follows is a worked example that really dives into the heart of where your head needs to be in using the smart technologies that exist today
I’ve previously written some of my thoughts related to Data Integration and Digital Strategy, however, perhaps I haven’t made it absolutely clear what I mean.
Digital Strategy is an enabler to allow you to tackle problems, however, unless you have a clear vision for the required data integration that follows, it doesn’t make a difference what your strategy is. What follows is a worked example that really dives into the heart of where your head needs to be in using the smart technologies that exist today
I’ve created a companion video that allows you to see this in action. The focus to have a user managed data integration layer is just awesome in my mind and allows for a seamless integration experience for a major project.
Commissioning Test Sheets
Commissioning Systems View
Here is a typical commissioning report. We are looking into the projects key systems. We see the number of subsystems, number of tests and total completed for each system.
We can follow the link on system to view what will will be subsystems
Commissioning SubSystem View
In our subsystem view we are obviously looking a bit in dept, and as we can see, this is our Integration Level. Each Subsystem/Disc pair is linked to a P6 Schedule ID. This is integration. However, we really need to dive into this and understand the process. We have 3 groups
Project Controls
Commissioning
Construction
The commissioning team will be updating our commissioning database, usually by close interfaces with engineering to define all the tests. They will be the hands on users of the commissioning system.
The project controls team will own the schedule, and will be responsible for updating and maintaining the schedule.
The construction team will know the detailed sequencing and duration that would be required.
Obviously each team will have discussions related to schedule and scope, however, your commissioning database is likely NOT updated with P6 IDs. Additionally, and here is the rub, when things change and perhaps more detail is added to the schedule, what is the mechanism for those changes to filter into the schedule. And when the magnitude and status of actual commissioning work changes, what is the mechanism for that information to be visible to the project controls team (in an smart way)
In the above example, all commissioning activities are assigned to 1 schedule ID. This is quite common when schedules are initially developed as projects enter execution. However, when additional detail is needed, it becomes hard to ensure systems STAY INTEGRATED.
Updated Data Integration
SubSystem EDIT MODE
The biggest weakness I see in many data integration efforts is they do not incorporate the step above. They do not allow edit capabilities into the data integration layers of the project.
In our worked example, we want to update the mapping between subsystems to schedule ID. We have added detailed activities into our schedule (I have left that step off, but imagine you have updated your schedule with additional commissioning activities), but do not yet have that new schedule information available in our commissioning system.
Enter Edit Mode and the world is your oyster. In the above screen we can now update the schedule ID mapping for each subsystem.
Updating the mapping to ScheduleID
Now that we have updated our mapping between our SubSystem / Discipline pairs, we can exit our edit mode and now see what we are left with
Now this is what proper Data Integration is all about. We have empowered our users to be able to flexibly manage their schedule, their commissioning system tree and update the mapping between the 2. Not to mention having everything at the fingertips of everyone involved in the process.
Extensions – JIRA
When we talk about Digital Strategy, we want users to be engaged and empowered to communicate. In the examples above, hopefully it was hard not to notice the “Jira Task” link that was available on the pages. Each of the subsystem discipline pairs is a critical discussion topic and as such deserves a Jira task page. This allows the team to openly discuss requirements and status. Indicate blocking tasks and interfaces (both preceeding and after completion of the task) is vital information.
Again, we want our users empowered to manage and communicate information on the project. This example really blows down barriers that I have seen. It is my passion to push the boundaries in the way we execute project. It is my passion to engage people and extract their knowledge as best as possible. Never before has technologies been so readily available to achieve these ends and I am really looking forward to seeing our business embrace some of my passion.
