Acknowledgement: Support for this project has been provided by the Australian Government Office for Learning and Teaching. The views in this project do not necessarily reflect the views of the Australian Government Office for Learning and Teaching.
- Project overview
- Project background
- Project methods
- Project significance
- Project findings
- > Stage 1. Data audit
- > Stage 2a. Participatory design (discovery)
- > Stage 2b. Participatory design (design)
- > Stage 3. Development of open data app
- > Stage 4. Evaluation of open data app
- Conclusion & recommendations
- About the team
Overview of the project
This project addresses the benefits and challenges that arise when students are provided with ‘open’ access to the learning-related data collected and collated in their university. The project supports students to develop open data tools and techniques relating to their learning on undergraduate and postgraduate courses at Monash University.
The process of higher education generates a mass of digital data relating to students, learning and the environments in which teaching and learning takes place; usually in disparate, disconnected and restricted forms. These data range from records of student satisfaction, online participation and grades through to class sizes and demographics.
This project was designed as a response to growing calls for better use of data within higher education. In particular it explored the contention that so-called ‘open data’ principles and practices might provide enhanced opportunities for university students to make informed decisions about their learning. The project explored ways of accessing official data-sets, and the possibility of creating a powerful and socially-focused ‘simple’ software application that allows non-expert users to directly manipulate and interpret data. By combining previously separate sets of already collected and archived data in a university context, students could well be able to ‘add value’ to the ways in which data can be used.
Of course, the realization of these potential benefits in universities face a number of likely practical barriers, not least the heterogeneity and dispersion of data sources, the difficulties of providing data in interpretable and usable forms to enable sustained use of data over time.
It is also necessary to acknowledge the risks of such open access to data. The present project was designed to approach the teaching and learning potential of open data in optimistic but circumspect terms. The overarching aim of the project was to move beyond the ‘hype’ and develop realistic understandings of the opportunities and limitations of open data use within student communities and the broader context of higher education institutions.
This project addressed the benefits and challenges that arise when students are provided with ‘open’ access to teaching and learning related data that is being collected and collated in their university.
The project worked with undergraduate and postgraduate students to develop open data tools and techniques relating to their studies at Monash University. These activities were designed to relate to a set of wider debates concerning the growing need for universities to make better use of data, and the potential of so-called ‘open data’ practices in extending the use and usefulness of data amongst student populations.
This project offers useful insights into the current limitations of data within university contexts, such as issues of data quality, data awareness and data literacies.
The project investigated the realities of digital data and higher education.
With ‘open data’ processes and practices now being implemented across many different areas of society, the present project aimed to explore the benefits and challenges associated with the introduction of ‘open’ access to data related to teaching and learning within universities. In particular, the project focused on open data use amongst students and addressed the following broad challenge:
To what extent can providing ‘open’ access to university data allow students to re-configure and re-use data in order to address ‘real-world problems’ relating to their learning behaviours and educational decisions?
More specifically, the project addressed six distinct questions relating to this broad challenge:
1. What data related to teaching and learning exist within universities that could be made openly accessible to students? What issues surround the potential ‘open’ configuration of this data?
2. What forms of open data access and use can be developed within/for student communities? What practical improvements need to be made to existing data practices? What new data tools and techniques could be developed to ‘fit’ the needs of student communities?
3. How might student communities work with these open data applications and practices? How might different groups make sense of, evaluate and combine open data to then generate new learning-related knowledge? What barriers emerge and how might they be overcome?
4. What support is necessary from teaching staff and university authorities?
5. What outcomes might be associated with the use of open data amongst groups of students? How might student uses of open data support the optimization of students’ learning, the logistics of learning and social/emotional aspects of learning? What limitations and risks (e.g. technical, legal, practical) are evident in the use of open data by students?
6. How might these outcomes be sustained over time and between different cohorts of students?
The project was conducted from January 2014 until May 2015, and focused on:
(i) assessing the nature of data currently being generated within universities relating to teaching and learning;
(ii) supporting students in ‘building-their-own’ open data systems and practices;
(iii) evaluating how these open data tools and techniques might be used ‘in the wild’ of higher education contexts.
The project began by comparing data uses from the Faculty of Education with the Faculty of Engineering. Subsequent work with students was with two distinct groups within the Faculty of Education:
Group 1: Undergraduate students enrolled in Bachelors of Education courses [studying fulltime, on-campus]
Group 2: Postgraduate coursework students enrolled in Master of Education courses [studying full and part-time, on-campus]
Working in these Faculties and with these groups of students, the project was conducted over four stages:
Stage 1. Audit of potential data sources
In Stage 1 a series of ‘data audits’ were carried out in both Faculties in conjunction with key administrative and IT staff in order to identify the nature and form of data sources related to teaching and learning. Eleven in-depth interviews were conducted in total.
This auditing stage of the project resulted in the identification of data relating to each of the student ‘case’ cohorts that could potentially be used in an open repository.
Stage 2. Participatory design activities
In Stage 2 ‘Participatory design’ workshops were conducted with students from each of the ‘case’ cohorts (a total of two groups) over a series of workshops. Students worked cooperatively to design open data services and practices that met self-identified needs relating to ‘real world’ problems that they felt that they encountered as part of their studies.
These workshops resulted in the production of a number of designs for how student groups wanted to access and use the data. The project team acted as scribes for the drawing-up of the final designs.
Stage 3. Development of open data applications & practices
Stage 3 of the project involved the project team taking responsibility for ‘building’ and beta-testing simple versions of the open data applications designed in the workshops. The project team, a consultant and three application developers from the Faculty of Engineering met regularly over a five month period to work through several iterations of the software development.
To facilitate sustainability, ownership and maintenance, the project attempted to avoid complex application development in favour of the development of software tools that were already licensed, free or open source and cloud/network based hosted. This stage was also compromised by a substantial amount of development time needing to be spent on the ‘cleaning’ and synthesising of different data sets to make them operationalisable.
Stage 4. End-user evaluations
In the final stage of the project, successfully developed applications were implemented and evaluated with the student participants during the following academic year. From this the project gained an initial sense of how these open data tools and techniques might ‘work’ in situ. During this stage both groups of students and staff were interviewed to develop a sense of how open data applications of this kind might fare ‘in the wild’ if they were to be used on a sustained basis across the student population.
This was a small-scale, exploratory seed project. As such its scope was limited, nevertheless, the project could be considered to be of genuine significance in a number of ways. In particular, the project has clear parallels with the growing interest in ‘big data’ and ‘learning analytics’ within higher education.
The present project is distinct from this previous work in five significant ways:
1. The issue of open data has been largely ignored in terms of university teaching and student learning. Conceptually and practically, then, the project constituted one of the first empirical explorations of these issues in the world;
2. This project was deliberately student-centred and student-driven. Whereas conventional forms of ‘learning analytics’ and data-mining are used primarily by teachers, administrators and other university authorities on an institutional basis, the present project supported the development of student directed and student focused forms of data use – allowing students to determine their own uses of data;
3. This project positioned students as active participants in the re-configuration of the data as opposed to passive recipients of institutionally produced analytics. Even when learning analytics is opened to students, it is usually limited to institutionally defined forms of aggregation, filtering and measures. This project proposed that students also need to be provided with opportunities to bring together personally meaningful sets of data, and apply their own configurations;
4. The present project was concerned deliberately with the re-use of existing data rather than the collection of new data and/or the monitoring of current learning activity. This project focused on the re-use and re-combination of already collected and archived data within different parts of the university organization. The project supported students to combine, aggregate and reconfigure already existing data sets in ways that were meaningful to them;
5. The present project, as part of exploring the applicability of ‘open data’ principles, included the goal of developing ‘really simple’ software techniques and tools in contrast with complex ‘analytics engines’, ‘datamarts’ and other data handling applications. The goal was to explore the use of simple applications and tools that students were already using and are familiar with. The focus of this project was on developing and supporting students’ effective uses of data, rather than the technical development of software.
While the end-product of this project was clearly limited in scope, it is perhaps worth first noting the relative success of the design cycles with both groups of students. Once statutory concerns over privacy and duty-of-care were satisfied, university and Faculty managers were willing to allow the opening up of selected datasets. At all stages of the project there was little overt institutional reticence or resistance to the general notion of open data principles. Subsequently, it also proved technically possible to design and develop a relatively powerful data-handling application for both student groups in the short timeframe of the project schedule.
The open data application and practices that were developed clearly added value and insight to these datasets, and saw a range of students collaborating and engaging in determining alternate uses of data that better served their experiences as university students.
While the substantive topic that these designs addressed was rather prosaic and bounded in nature, it nevertheless constituted alternate uses of data to address ‘real-world’ problems and allow students to consider ways of engaging with data that supported meaningfully oriented behaviour. From this perspective of proving ‘proof of concept’, the project could be seen as a qualified success.
The project findings are presented by stages as shown on tabs under “Project Findings”:
- Stage 1. Data audit;
- Stage 2. Participatory design;
- Stage 3. Development of open data app; and
- Stage 4. Evaluation of open data app
Stage 1 – Data audit
The project team conducted a ‘data audit’ in conjunction with key teaching, administrative and IT staff within the Faculty of Education, Faculty of Engineering and the university’s central services. This process was designed to identify the nature and form of internal and external data sources relating to student cohorts in use in each Faculty.
As with any aspect of administration and management, the actual implementation of these data procedures and processes was varied and often restricted. In this sense a number of institutional issues and concerns surrounding the use of this data found in this project are reported in Table 1.
Table 1. Challenges to open data in Higher Education
From the wide range of data collected within the university only four specific datasets relating to teaching and learning were identified by Faculty managers and central university staff as realistically suitable for open access and use (see Table 2).
Table 2. Identified potential open data sources
As is intended in any participatory design project, these activities were as revealing in terms of their ‘failures’ as any apparent ‘successes’. As such, there is much that our findings say about the limitations of the open data philosophy when applied in the realities of university teaching and learning. Perhaps most prominent was the limited ‘open data’ potential of the datasets that were being produced within the university and faculties.
Of course, the data was clearly working well in terms of the university administration and management. Yet as source material for any potential open data project, most of the data being produced in across the faculties and university related to relatively narrow sets of measures, such as officially defined teaching and learning ‘performance’ metrics and outputs, homogenised indicators of ‘well being’, and pro-forma unit information. This clearly constrained the possible nature and utility of any reapplications of this data.
In short, this was data with little ‘radical’ potential for re-use and re-purposing. Most importantly, the quality of these data-sets fell well short of what might be expected from well-organised and well-run data infrastructures – e.g. clean, interoperable data, regularly updated and archived with appropriate meta-data.
Stage 2 – Participatory design activities
During the first sessions of the participatory design workshop cycle the students and project team worked together to identify real-world problems associated with the four university data sets identified as suitable for open data re-use. While the data sets (as just stated) did not constitute ‘ideal’ examples of open data these participatory design activities still offered valuable insights into the potential re-use of data within the university.
Having been familiarized with the open data sets, the two groups of students discussed and developed a number of ‘real world problems’ that the data might address. As might be expected, these issues related closely to the respective ‘jobs’ of being an undergraduate and taught post-graduate student.
Thus undergraduate student suggestions for potential uses of the data focused largely on what was referred to as “survival” [UG participant] as a student. This related predominantly to obtaining satisfactory grades for different units, choosing “easy and fair” units that would not jeopardise grades; gaining a sense of course expectations prior to enrolments and managing workload and deadlines.
Similarly, the post-graduate group highlighted potential uses of data that were orientated practically towards issues of gaining a better sense of the Monash context and an accurate sense of their own progress. As with the undergraduate participants, the desire to access data relating to performance in terms of assessment grades was stressed throughout the preliminary workshops.
the potential of open data for both groups strongly resolved around being more informed about how to succeed in assessment tasks. This orientation towards performance, in contrast with ‘learning’, was highlighted when both groups of students dismissed any usefulness of ‘learning analytics’ data that might be available through the learning management system.
Both workshop groups therefore made sense of the four data sets in terms of three main potential benefits:
1. expectations of clarity, the information that they would most value being given access to when making choices about which units to take;
2. the data sets having the potential to provide a better sense of context – be it in relative terms of institutional expectations, and their own performance in relation to the performance of others; and
3. the potential of data as a source of insight
Stage 2 – Participatory design activities
Given the broad issues and interests, both workshops went on to develop similar sets of designs for data tools and techniques. These related broadly to tools that might distil the data sets for human judgement and/or make use of the data to analyse trends and patterns. Both groups’ designs therefore related to tools of description and explanation of the data, rather than for more complex processes of automated prediction or prescription.
As such the workgroups resulted in the development of five distinct forms of relatively basic data application:
1. Participant interest in designs based around what was termed a ‘data dump’. These were designs to simply present all the data available on an individual, unit or course cohort;
2. A desire to ‘see more’ involved designs based around the reduction of data – what was termed a data ‘synopsis’ or ‘overview’. For example, data ‘filters’, the concept of a “dashboard”, simple graphic representation of data (e.g. bar charts, box-plots and word cloud generators);
3. Data tools that offered a sense of ‘comparison’, ways of presenting correlations between different data fields (for example, grades and type of assignment task). Other recurring design features included tools to identify ‘trends’, ‘patterns’ and changes over time on a whole class basis;
4. Data tools that offered a sense of ‘individual tracking’. Here workshops explored ways of presenting data relating to the monitoring of individual performance, and expected outcomes; and
5. Finally was the more ambitious design of data-based ‘notifications’. For the undergraduate student workgroup, this centred on using data to be informed of expected work, deadlines and other impending events
After identifying these sets of real world problems, and exploring their applicability to the available data sets, the undergraduate and postgraduate groups then engaged in a series of exercises in which they described, sketched, and ‘stepped-through’ how specific data could be used to meet their needs. This was facilitated by the project team, which also included one of the application developers (see Figure 1).
Figure 1. Postgraduate design workshop
It is noteworthy that both groups of students eventually focussed on the use of unit climate data (e.g. previous cohort grades and student satisfaction reports) and unit details (e.g., details of teaching staff, assessment criteria, and so on). Despite encouragement to imagine more expansive possibilities in using this data, it became increasingly clear to the students and the project team that the data available was both limited in granularity and scope. This, in effect, constrained the re-purposing of the data.
While constrained in this manner, both workshop groups produced a number of design sketches that were broadly felt to address some aspect of their identified ‘real world problems’ and preferred forms of data use (see Figure 2).
Figure 2. Sample paper designs of prototype components
Due to the similarities across the designs, it was decided that the development team would create a single application that addressed the needs of both designs (rather than dividing resources to create two separate applications). The final paper prototype that combined the design components from the two workshop groups can be seen in Figure 3.
Figure 3. Prototype sketch combining both undergraduate and postgraduate designs
While the paper prototype was successful in articulating ideas between the students and the project team, it was not sufficiently detailed to act as a design brief for the application development team. Therefore, the project team engaged in a process of backward mapping, exploring workshop recordings, sketches, and observational notes to develop an elaborated design brief which contained three application proposals. Table 1 presents an extract of the final design brief provided to the development team.
There were obvious limitations in the data skills and ‘data imaginations’ of the students who participated in the design workshops. While clearly interested and engaged in the activities, neither of our workshop groups possessed adequate programming skills, statistical knowledge or analytic capabilities to reimagine sophisticated ways of working with the datasets. Similarly, most participants’ ‘data awareness’ (or what could be termed ‘data literacy’) was confined to very restricted modes of data engagement – most notably following the simplistic forms of data visualization and data-handling familiar from popular online applications and services. There was little interest and capacity for more advanced engagement with the data practices beyond star ratings, box plot graphs, dashboards and filters.
Stage 3 – Development of open data application
The aim of this stage was the development of ‘really simple’ applications rather than fully working products. This stage allowed for exploratory investigations of the likely applicability of the open data philosophy in the realities of the university context. As such, the development of the data application raised a number of salient issues with regards to the technical/programming feasibility of the workshop designs; the limitations of the open data sets; and the likely practicality and use of the data applications.
The development team used their experience of participating in the workshops, the paper prototype, and the design brief to develop the application. The process was lengthy, largely due to the complications of gaining access to some of the data sets in a suitably ‘raw’ format. These complications related primarily to faculty checks and permissions regarding privacy and risk. In addition, as well as the format of the provided data had significant implications for the development team’s ability to respond to the design, and in the functionality of the application.
Open data projects are best served by the development of ‘simple’ tools using familiar software such as Microsoft Excel or Google Sheets. Such an approach makes any such tool more likely to be further maintained and extended by the users. However, after considering a range of options, the development team judged that these simple systems were not the best choice for the designed prototype as they would limit what could be achieved with the data – in particular how it could be displayed and interrogated as per the design brief. In order to make the visualisation and manipulation of the data intuitive for students who were not familiar with data modelling in programs such as Microsoft Excel and Google Sheets, a custom interface would need to be programmed. In addition to this, the design briefs involved live searches of media such as YouTube and LinkedIn in order to provide additional details about the subject and/or teaching staff. Finally, the workshops also highlighted the value of a system that could be used on a mobile or tablet. As a result the development team chose to develop a web application that interrogated a database that was maintainable and scalable through automated procedures.
Figure 1 illustrates one of the application functions that to allows users to filter and compare Units (subjects) according to a star rating based on student evaluations.
Figure 2 provides an example of how students could interrogate the data relating to each of the Units (subjects) including the unit description, assessment, grades distribution, student evaluations, and even searches of unofficial sources such as LinkedIn and YouTube.
Figure 2. Screenshot of subject level function in the data application
(Note: the actual unit code, unit name and lecturer name have been hidden for deidentification purposes.)
There was the likely limited sustainability of these open data interventions beyond the involvement of the project team. It should be noted that completing even the one ‘simple’ open data application described in this report involved hundreds of hours of organization, drafting and programming work. Even with the mundane instances of opening access to student satisfaction data, or visualizing grade data over time, the labour-intensive nature of our project illustrates the large volume of work that lies behind the abstract promise of ‘setting data free’.
While the programming aspects of our project was intended to be relatively ‘simple’, technical development of the working prototype was restricted by constraints relating to the nature and quality of available data-sets. These were not straightforward processes of connecting data compiled at different locations together.
Instead, the project faced considerable variation in the quality of the data that was accessible. Put simply, the open data application that resulted from our project were sustained by a great deal of ‘behind the scenes’ technical work by the research team.
Stage 4. Evaluation of the open data application
The final stage of the project involved the developed open data application being presented back to the student groups in order to gain a sense of how these open data tools and techniques might ‘work’ in situ. Follow-up testing sessions and interviews were conducted with both groups of students to develop a sense of how these open data applications might fare ‘in the wild’ if they were to be used on a sustained basis across the student population.
In general, both groups were positive about the developed application in terms of:
- its applicability to their future unit choice making;
- the range of data representation options was welcomed as catering for different preferences and numerical abilities;
- the ability to make more detailed comparisons about their own performance in relation to others
The project found there was limited ‘democratising’ effect of the open data design activities and applications in universities. Indeed, while our projects could be said to have provided a ‘proof of concept’ with regards to the technical aspects of procuring and programming open data interventions, it could not genuinely be said to have realised the main claims of open data – i.e. improvements in engagement, empowerment, democracy and accountability.
The design and development of our application might have been of practical use within the specific context of the Monash Faculty of Education, but it was hardly challenging the institutional status quo within either the faculty or the wider university. Instead, this was a data application that fitted well with pre-existing power relations and conditions of teaching arrangements. This is perhaps the most significant ‘non-finding’ of our research – i.e. the largely unthreatening, non-controversial, non-transformatory nature of the open data tool that resulted from our participatory design activities.
This project marks a first attempt to test the aspirations of open data against the organisational realities of the contemporary university. Clearly any attempt to support more ‘open’ forms of data access and use within universities faces a number of potential challenges. These include: the quality of the data itself; the congruence between the goals of open data projects and the concerns and responsibilities of the higher education institution; ‘external limitations’ such as ethical, legal and managerial concerns; and ‘internal limitations’ such as the data skills and competencies that exist throughout student and educator communities.
In short, our research found that while the democratic potential of open data may have been overly optimistic, we would not conclude that university educators and student groups should give up on the ideals of open data altogether. Yet this does suggest that any promises of an imminent ‘open data revolution’ are acted upon in a critical and circumspect manner, with any attempts to ‘open up’ data within universities approached with suitably tempered and realistic expectations.
This project has tested the feasibility of open data approaches relating to student use of university data. We are now in the position to offer a concluding guidelines and recommendations highlighting the practical limitations of open data principles and philosophies – i.e. working through institutional concerns over appropriate data use; recognizing the likely technical difficulties of such data work; as well as working to develop and sustain student capability and interest.
An open institutional approach…
- recognises that open data projects are necessarily about students’ real world problems – not about institutional problems;
- recognises that open data projects involve considerable and ongoing commitment of time and resources;
- adopts a flexible approach to imagining and designing what data may be used for;
- is comfortable with a bottom up approach to development;
- has a realistic expectation of the likely limits of any open data application that is eventually developed;
- recognises that data access and privacy are key mitigating issues that need to be debated and discussed at all levels;
- recognises the limited capacity to repurpose data that has been collected for specific institutional purposes.
Open data work and application development…
- involves a considerable amount of planning to collect and collate data that meets the principles of open data (e.g., raw, live, interoperable, etc.);
- involves a considerable amount of technical work involved in making data usable and re-purposable;
- values functionality over form – that is, issues of aesthetics and ‘completeness’ are outweighed by the drive for addressing a real world problem.
Open data projects…
- rely on a fully participatory approach – based on partnerships between central data experts, application developers, and students;
- centrally position student voice in terms of identifying real world problems and meaningful ‘solutions’;
- trust students to make sensible and meaningful decisions with regards to their own needs;
- support students to develop data literacy and data ‘imaginations’ so that they can be fully involved at all stages.
General links about open data
- What is Open Data? – The Open Data Handbook
- Open Knowledge: What is Open?
- Open Government
- The Open Data Foundation
Libraries for Data Visualisation
Tools and Libraries for Creating Data Servers
In order to provide data to multiple, internet-connected devices, a server is needed. This data server can also be used to serve the files that make up the graphical user interface and logic of the web application to the users’ phones. We wrote the server application in Express JS (which utilizes the Node JS programming environment) and MongoDB for the database.
Resources for Creating Web Applications
Resources for Parsing Data
Before data can be displayed, it will often need to be manipulated into some form of standardised data structure and file format. This generally involves reading from one file format, extracting the useful information, structuring this information and then writing it to another file format. The following tools and resources can make this process much easier.
- Documentation on CSV file reading and writing (for Python)
- This Python script (for converting xlsx to csv)
- Numpy library (for handling big data – particularly arrays – in Python)
Recommended development environment
As with all software projects, some form of text editor is required to write the code. In this project we used a combination of Vim and Sublime Text. Unless the user has some experience with Vim, we recommend using Sublime Text or similar.
Note: The primary applications used in this project (MongoDB, Express, Angular and Node) are often used together and, as such, they have been nicknamed the “MEAN” stack. The Mean JS initiative packages these technologies together and may be a good starting point for similar projects. A tutorial describing how to create a simple “MEAN Stack” application is described here.
Neil Selwyn is a Professor in the Faculty of Education, Monash University. He has worked previously at the Institute of Education – University of London, Cardiff University and the University of Bristol (all UK). Neil’s research and teaching focuses on the place of digital media in everyday life, and the sociology of technology (non)use in educational settings. Neil has written extensively on a number of issues, including digital exclusion, education technology policymaking and the student experience of technology-based learning. He has carried out funded research on digital technology, society and education for the Australian Research Council (ARC), Economic and Social Research Council (ESRC), British Academy, the BBC, Nuffield Foundation, the Gates Foundation, the Spencer Foundation, Microsoft Partners in Learning, Becta, Centre for Distance Education, the Welsh Office, National Assembly of Wales and various local authorities.
Michael Henderson is a Senior Lecturer in Information Communication Technologies in Education at Monash University. He has provided expert advice to government and other education providers in the field of eLearning and delivered many keynotes at conferences with the aim of expanding delegates understanding of the opportunities afforded by educational technology, both in online and in-class environments. Michael was the leader of the Monash Centre for Educational Multimedia (CEMM) which has transformed into the Learning with New Media Research Group in which he continues as a key member. The group provides leadership in rigorous empirical research for the advancement of our understanding of teaching and learning and the various roles of technologies. Michael has been an executive member of AARE and a state councillor of ICTEV and is the co-editor of the Digital Learning and Teaching Victoria Journal. Michael is an associate editor of the Australasian Journal of Educational Technology.
Shu-Hua Chao was a primary school teacher who became interested in research. Her expertise lies in the field of ICT in education and female participation in STEM. She has worked and conducted research in both Australia and Taiwan enabling an international, and cross-cultural lens to her work. Shu has taught postgraduate and undergraduate studies in educational technology, instructional design, learning theory, and educational psychology. Shu is also a skilled project manager and research fellow, having worked on funded projects from the Spencer Foundation and the Australian Office for Learning and Teaching.