Bringing chart-making out of the back-end

>> DANIEL COTHRAN:  Thanks, Andy.  

And thanks to all the MidCamp organizers.  

And for everyone who's logged in.  This is bringing chart‑making out of the back end.  

I'm Daniel Cothran, Drupal.org.  

Before this year, I frequently had to explain what public health and epidemiology are.  Sadly, that's no longer the case.  

As you might have seen, there are a lot of public ‑‑ excuse me ‑‑ a lot of challenges issues in public health.  And we have a lot of projects working to address many of these issues.  

Some of those projects had Drupal websites and wanted to visualize their data.  And I started working on a Drupal 8 version of the charts module, and I've since become the primary maintainer.  

So let's get started.  

Count how many times you've thought about Charts in the last few weeks.  The phrase flatten the curves has been tossed about so many times it's been memed.  As you can see on the slight, with the "cattening the curve."  

I can tell me why data visualize is important but that would be a waste.  You know.  You're here.  What I do want to say is this:  It's time for data visualizations, proper accessible data visualizations, to be more routinely included in our content management just like images or video.  For that to happen, we need to make it easy for every day users to create their own charts.  

And because Drupal feels like home, you know, that place we're all stuck, I want people to be able to create these visualizes without needing to leave Drupal.  

By the way, Drupal, the community, was talking about flattening the curve years ago.  

In this presentation, I will walk you through a few scenarios involving data visualize and show you how the Charts module can help with each of them.  Are there other technologies that could also do the job?  Yes.  Will I be covering them?  Mostly not.  If you're hooking for the consumer reports of charting technology you will give me a filled in black circle at the end.  Presentation.  A jock for my Mom.  

I'm doing everything I can to make it a better, more useful module, including by spreading the word and trying to bring others into the fold.  All that said, I would mention a couple of alternatives at the end.  I also hope to hear from you.  What are you looking for when it comes to data visualize?  Please be thinking about that.  

With that, let's dive in and look at some scenarios in which you may find yourself.  

You have the Charts installed and a library in place.  I'm assuming you can install additional modules or you have someone who can help you do that.  

So scenario one is, I want a dashboard.  

My company being a public health company following links about COVID‑19 was not something we did during lunch break.  We also had to data this listserv and that's where I became acquainted with this ‑‑ [indistinct] ‑‑ data pack.  My first thought, I could build this dashboard using the Charts module.  

A data dashboard is a common ask at my company.  Last year when I presented, I finished a dashboard on tuberculosis data the day before and build them tracking the roll out of a contraceptive, tracking indicators around project performance.  

The data and the TB dashboard on the slide are from the WHO, release the data in CSV files each year for publicly at least a decade.  My colleagues have been studying TB and monitoring health systems for years, but they found something worth investigating nearly every time we look at the at these charts together.  

Charts makes it easy to build dashboards within Drupal.  Option 1, to use Layout Builder and Charts blocks or views.  

I was slow to jump on the Layout Builder train because I was using a theme that didn't work well with it.  When I started using it, it was a solid option.  It's built into core so that's a great bonus.  You can set up layouts for a whole bundle, a content type, or on a per entity basis.  

While I was preparing for this presentation, I did ‑‑ I did find a few draw backs.  First, the settings tray is too narrow.  It can't contain the data entry that comes with the most recent version of Charts.  And I need to add alignment CSS for the charts to say in the same row.  That maybe unique to Layout Builder users, and I think it's an easy bug to fix.  

So step one for this is to create a dashboard content type.  Or at least that's how I would do it.  

So you can see that in the picture.  

The next step is to manage the display of that content type, and enable Layout Builder by checking the boxes that you can see toward the bottom of the screenshot.  

From there, you would create a dashboard node.  And then after you saved it, you would go in and adjust the layout.  

And one of the things you could do is click add section, and then add a layout to that section.  In this case, I have a three‑column grid.  

Once you have the layout in place, you have the option for adding blocks to that layout.  These can be blocks that come from a custom block type or they can be blocks generated by you.  You can insert fields here.  

And my case, the next slide, I have added a chart block.  So one thing that I did do with this is I manually adjusted the width of the settings tray so I had more room to enter in the data that you can see in the table display, using inspect element.  

So I added three different charts, and then I saved the layout and went back to the page.  And you can see them hear in a row, just like this.  

So you see, it was fairly simple to do.  And as I was mentioning, you can include various types of charts.  Charts that are made as blocks, charts that are fields on a content type, or charts that are coming from a view display.  

And if you use those, I think those are particularly well‑equipped for dashboards, because you can have contextual filters.  In other words, some sort of argument up in the URL that allows you to filter per page.  

You can also add exposed filters to any of these view‑based charts.  

The second option for building a dashboard is to use Paragraphs and Charts fields.  And you can actually use Layout Builder for your Paragraph display if you want to.  Because with Paragraphs you're generally working within you the node edit page it offers more ‑‑ this may also be a more familiar way of adding content for long‑time Drupal content managers.  

Because you're setting up Paragraph entities, there may be also be more ‑‑ [indistinct] ‑‑‑ability and reusability by going to go with this option.  But the likelihood that you would have more than a few design patterns seems low.  A potential withdraw back to using this option, Paragraphs, a depiction, contributed module.  It can work well but be hard to manage, especially if you have a multilingual site.  

So for this option, step 1 is to add a paragraph type.  

And in that paragraph type, you would want to add at least one chart field.  Which I have shown on the screenshot.  

Then you would add a content type set up to use the paragraph, and I would suggest potentially having the option of adding multiple instances of that paragraph.  

Then you would create your node, as and as you can see in the screenshot at the bottom, there's a Charts paragraph, and you can see the chart field with its setting form there, which is where you would enter into your data.  

Once you have completed the node edit page and you safe your node, this is what the front end might look like.  In my case, I have a text area field on the left and a chart on the right in each paragraph.  

Okay.  

So the second scenario that I wanted to go through is, I want to display the results of a Webform survey.  

The Drupal community has an amazing tool for surveys, the Webform module.  There are many good reasons to have users submit a Webform.  

A nice feature in certain circumstances is to display the users results back to them after submission.  

Either just their results or the results incorporated with all the other results.  This can encourage users to submit, especially if we're talking about routine data collection, which is an important tie data collection in the public health field.  Charts is a great way for users to see their input is important and to encourage participation and routine data collection.  There are at least a couple of ways of generating charts with Webform results without the need to write code.  

I almost always use views to construct the charts on my site.  So when I tried to plot Webform results using the Webform views module would be my first instinct.  The great thing will this option is that you can keep your data within the Webform results table and you can build charts how you normally would.  But you have to be thoughtful about what you're looking for in order for this to be a viable option.  

The Webform views module has an unpleasant naming convention instead of the field name.  You get Webform submission value, submission value 1, 2, et cetera, and it can be a little less user‑friendly.  

Hmm ...  

I'm not sure what's happening here.  

Sorry about this.  Can you all still hear me?  

>> Yes, we can still hear you.  

>> DANIEL COTHRAN:  Thank you.  

Okay.  Let me try the screen sharing again.  

[Computer noise.] 

Okay.  I don't know why it doesn't like this slide.  

So I'm going to ...  [muttering]. 

And just exit out of the presentation.  

I apologize for the technical difficulties. 

>> ANDREW OLSON:  No worries.  Give it another try.  I downloaded just downloaded a presentation.  And your slides are up on the mid camp site, so I have your slides local.  Worst case, you can do something else.  

Keep on ‑‑ 

>> DANIEL COTHRAN:  Keep using my screen? 

>> ANDREW OLSON:  I see the slides layout.  

>> DANIEL COTHRAN:  I may keep it here for a minute.  

The first step is to create your Webform.  

And with your Webform, add at least ‑‑ well, you don't have to add at least one number field.  But typically your number fields are easier to plot.  You don't have to necessarily rely on aggregation.  

And then you'll want some results in area Webform, so I've added a couple ‑‑ just dummy data, and the way I have this set up, there's a text field for the name of the module, and then three number fields for the 7, 8, and 9 versions of that module.  

So I've completed this twice in my little demo.  

And then you would create a view, and add your different Webform fields to the view.  

You would set the display format to be a chart and configure the settings for that chart.  If you look at the screenshot, I've made the module name the field, and the number fields for 7, 8, and 9 versions the data providing fields.  

And when you go and you preview, you'll see you have ‑‑ in this case a column chart.  And the way it's done here, it's clustered by submission.  The first three bars that you see were 7, 8, and 9 version of the Charts module, and then I did it for views database connector as well.  

Another way that you could do this, visualize this same data, would be to stack the chart, which would allow to you fit in more Webform results into the same chart.  

So a second option for creating charts based on Webform results uses Webform content creator to turn these results into nodes, which can then be queried in views and displayed as charts.  This is easier to query and use than the Webform admissions table and it gives you more flexibility when it comes to your data structure.  You can turn a single Webform submission into multiple nodes.  Drawbacks to this reach, you have to set up the connection between the Webform submission and node.  You have two copies of the same data.  The opportunity for something to go wrong in the process of node creation, and the contributed module that's fairly new and not mature.  It's not covered by the security policy and only has approximately 200 users currently.  

So again, with this one, you would start by creating your Webform.  

And then in this case, you would also create content type that would receive the Webform submission data.  

Then you would create a Webform content creator entity, which allows you to select the Webform to use in the content type to end the results to.  

And then you can configure that Webform content creator, and ‑‑ or the fields from the Webform into the content type.  

Because I had a number value for 7X, 8X, and 9X, I ceded three different Webform content creator entities, generated three different nodes per Webform result.  

So I'm showing you again the same Webform basically, except this one was designed for Webform content creator, and when I submit this; then go into the content screen, you can see there are ‑‑ I submitted it twice, and so there are six results, three for Charts and three for Views database connector.  

So the next step is to add a view.  Edit that view, configure it to use the chart format.  

And in this case, I decided to using aggregation, and I'm showing the version as the label, and the downloads field ...  and ...  

I'll show you the next screen.  That's the preview of that result, which I made a pie chart.  

So another scenario that you might find yourself in is, I want anonymous users to generate charts.  

In this case, I'm only going to give you one option.  And that is using views fields on/off, and the exposed chart type modules.  Exposed chart type is bundled in.  It's a filed that's bundled in with the charts module, and views fields are been ‑‑ [indistinct] ‑‑ module that I ported to Drupal 8 along ‑‑ [indistinct]. 

So the first few steps are to enable that module, and then edit your view that's using a chart display, and then configure the on/off form field, which you can see at the bottom of the field list there.  I don't have the actual configuration screen open, but it lets you choose other fields within the same view to be able to toggle between.  

I also configured the expose ed chart type field, a similar layout to the fields use on/off field.  And I've selected the chart type options that I want available, and the selection type.  In this case, it's a single select.  

I saved my view and opened a new browser and went to the views page as an anonymous user.  You can see there's a chart, but it's got a lot of data, and it's not necessarily going to be that helpful to me.  

However, there are all those filters at the top.  

And so an anonymous user could then apply some of those filters, as you can see in this next slide.  

And I just wanted to point out a couple different things.  The first filter is chart title, a custom module that I created that allows to you write the title that you want and put into the chart.  

So you can see that that's applied.  

The next is the exposed chart type.  Now you're seeing a column display the chart.  I selected both the award value in US dollars and the population fields to be displaying.  I picked a location to narrow it down, the results.  

I picked location, and award transaction category, or both taxonomies that are on the content type that we're displaying with this view.  

And I also have an exposed sort here, which I'm not using.  

And in this case, this is using the high charts library, which has options for downloading this developed chart as an image, or as a data table, and so someone could take this and add it to their own report or something like that.  

So I mentioned earlier that I would talk about a couple other charting resources.  The first one that I wanted to mention is DKAN, open data platform.  And this is still a Drupal distribution that has really powerful charting capabilities, and sort of data storage and everything.  

The draw back to this, it is a distribution.  And it has sort of a specific setup.  

So you're starting from a particular information architecture.  Whereas with the Charts module, you can basically take any ‑‑ any Drupal website that you have and install the Charts module and then visualize data, sort of however you want.  

The next thing I wanted to point out, there is a CK Editor plugin for creating charts.  

It's fairly limited in features, but it's extremely easy to use and it's great if you're wanting to add charts into text area field.  But if you have a view or things like that it's not going to enable you to make charts that way.  

The final thing I wanted to mention is the High Charts editor.  

Which I haven't used myself, but just reading about it, it seems like it has a lot of great features, and High Charts is an extremely powerful charting library, which is one of the ones that the Charts module is able to make use of.  

So that's the end of my formal presentation.  You remembered earlier that I asked you about different scenarios in which you might want to have charts.  

So now would be a great time to talk about that.  

And I'm happy to field any other Charts‑related questions.  

With that, I'll go ahead and open it up for questions and comments.  

>> ANDREW OLSON:  You can put those in the chat.  If you raise your hand, I can also unmute you, and you can talk directly.  Either one works.  

There's a question, Fredric, I'm going to try and find you and unmute you.  

>> How is it going?  Can you hear me?  

>> DANIEL COTHRAN:  Thanks for your question.  

>> No problem.  So, yeah, I saw your presentation.  I've watched videos before.  I'm still trying to wrap my head around is best practices in terms of preparing for the data so we can properly ‑‑ the use case that I had was ‑‑it was a D7 type, but I mean, I know you've done D8 stuff, and I'm ‑‑ I'm creating that site.  But it was a D7 site with CBCRM, and I know the way that data is stored is not the best.  But I was curious, you know, as I would have just ‑‑ best practices, is it, you know, to ‑‑ I mentioned you said, hey, make it a number feels so you don't have to use aggregation.  That seemed to be an interesting nugget there.  

Are there other things when it comes to kind of making sure that you can do your charts, but one ‑‑ I guess the specific scenario I'm thinking of is trying to combine multiple data sets onto a single chart, so that they can be compared, so ‑‑ I know you had some examples there.  

Of, like ‑‑ of different types.  I'm curious, is, like ‑‑ is it just making sure it's a number, and that the values are numbers, is kind of, like, the big ‑‑ the big thing, to make sure all it works, or are there other things?  

>> DANIEL COTHRAN:  That's a great question, and actually, sort of brings me to another couple of talking points.  

Yeah, I would say just in general, numeric fields are going to be the easiest to chart.  You can also, if you do aggregation, and ‑‑ and then do a sum, or a count, it should be able to read that as well.  

But generally, if you're setting it up in a way where it's reading the number from one of your fields and then using a label from something, that tends to work best.  You can do that aggregated and unaggregated.  

I, generally, am building charts based on views.  And as you are probably aware, views is really great if you have, like, 50 different things you want to show or less.  And once you start to get into, like, 100, 200, 300, it can take quite a while for the view to be able to render, because it ‑‑ it's trying to pull in each one of ‑‑ let's just use content types as an example.  

It's trying to pull in each node in order to, you know, grab the data from it, and display it.  So if you're trying to do something that's really data‑incentive and you need thousands of data points, Views wouldn't be a good solution.  Unless you do one of a couple different things.  

So one option is there's a module called Denormalizer, which a ‑‑ I can't remember if it was at the last ‑‑ the Washington DrupalCon ‑‑ I think it was DrupalCon.  There was a presentation on the Denormalizer, and now there's a Drupal 8 version of it, and it allows you to, instead of the way the database is structured now, where you have multiple different tables that sort of comprise your node data, it allows you to create basically a flat table where each row is going to have, like, your node ID and all the values of your different fields on that content type.  

And you can connect your view to that denormalized table.  And instead of having to load your node, it pulls directly from the row of the denormalized table so it can pull data a lot faster that way.  

Another option is if you have a ‑‑ either a database table or a remote database, use the views connector module which I mentioned earlier, and it pulled directly in the same way.  Straight from a table to.  It doesn't have to do the entity loading, which is what makes Views so slow.  

You can also, if you have your data in ...  like some other format, and you have a developer, use the Charts API to be able to create a chart.  

The other thing I would say is:  Certain modules ‑‑ I was going to present this, but then I thought my presentation would go too long.  And it looks like it sort of is.  

Drupal commerce ‑‑ 

>> ANDREW OLSON:  A few more minutes.  Yeah.  

>> DANIEL COTHRAN:  Thanks.  

Drupal commerce can be the price fields ‑‑ I've had a little bit of trouble with sometimes if I want to use them with Views.  

And so I created a little sort of helper module called Commerce Views Connector, and it has a method that use views to get the value of a field.  And that makes it easier for Charts and for some other modules like View simple math field to get the value of the actual ‑‑ the number of your price, and allows you to chart that.  

I hope that was helpful.  

Is it okay if I take at least the next question?

>> ANDREW OLSON:  Yeah.  We'll ‑‑ we'll try and stop at 2:35 year.  But I see Jess has a question.  That what you're going to take?  

>> DANIEL COTHRAN:  Yeah.  

>> ANDREW OLSON:  Great.  

>> DANIEL COTHRAN:  So organization charts, I have not tried to use with the charts module.  

I think just from what I've seen of them, they can be pretty complicated.  

And so I haven't really seen a good way to make, like, a standardization there.  I think that you probably could relatively simply use the Charts API to generate organization chart.  

I do have an example, not of an organization chart but of using Google geomap ‑‑ I think that's what it's called ‑‑ or geochart, which allows you to have, like, different gradients of colors on a map, and that I was able to do pretty simply using the Charts API and really just minor code.  

>> ANDREW OLSON:  Two minutes left.  There's one question from Jason.  "Are you aware of anyone using this to chart data obtained via a restful web service call?  The call being made from Drupal, not the front end."  

>> DANIEL COTHRAN:  I am not.  

>> ANDREW OLSON:  All right.  Well, I want to thank everybody.  Once again we will be getting another session in this room.  But thank you so much.  Feel free to continue the conversation in either MidCamp 312.  You can take it to the MidCamp Slack general hallway and start a thread there.  Please, please, please, provide feedback on the MidCamp site for this session.  

And we will have caption ‑‑ sorry ‑‑ we will have a transcript of this up on MidCamp site at a later time.  Thanks, everybody, for joining.  And have a great rest of your day.  

>> DANIEL COTHRAN:  Thanks.