Olly and Liam got to grips with linking ArcGIS and MySQL server last week, essentially they have created a method to allow ArcGIS to talk directly to the HALOGEN data without the need to export data (e.g. as a csv or tab delimited text file). So far it looks like we can query directly on the database. Why is this great news you ask? Before I created an events theme in ArcGIS which was then converted to a shapefile, unfortunately with large volumes of data we exceeded the maximum size of shapefiles and therefore could not query the data any further without a crash. We still need to test and see if relational databases work though, watch this space...
Andrew.
Showing posts with label jiscgeo. Show all posts
Showing posts with label jiscgeo. Show all posts
Monday, 8 August 2011
Tuesday, 2 August 2011
Plotting aggregated points over Google Earth satellite images
Olly has created a web interface that plots our data on to Google Earth satellite images. Our data is aggregated to the centre of BNG 1km squares to preserve confidentiality and to standardise the resolution of our database as there are several different data sources to compare. I am concerned that end users may forget / not read the project documentation and think that a point marks the exact location of data when in reality it could be anywhere in a km square around the point. Does anyone else share these concerns - or have a way of reminding an end user of this?
Andrew.
Andrew.
Tuesday, 5 July 2011
Progress Update for June
Good progress is being made and we have now delivered the new data sources as planned!
Data Sources
A key target was to complete the load of the new data sources to HALOGEN during June and this has been achieved. Well done Olly and Andrew !
The coverage of the PAS data has been increased to all of England. The Capelli data and 1881 Surname census data has now been added to the database.
The load of an additional source of surname data has been requested and will be added over the next few months.
The load of an additional source of surname data has been requested and will be added over the next few months.
Tools Development and Evaluation
Development of prototypes of the ‘data extraction tool’ (using Business Objects) and a ‘web based enquiry tool’ continues.
Development of prototypes of the ‘data extraction tool’ (using Business Objects) and a ‘web based enquiry tool’ continues.
A plan for the iterative development of the ‘web based’ enquiry has been agreed. The second iteration of development is now complete and the system was presented to users at Nottingham University's Institute of Name Studies on 28th June.
Other
A communications plan to support internal dissemination activity relating to the project has been drafted for Board approval.
We had our first Project Board meeting on 27th May and that went well.
Dave Flanders, our JISC Programme Manager has scheduled a visit for 2nd August.
Dave Flanders, our JISC Programme Manager has scheduled a visit for 2nd August.
Wednesday, 11 May 2011
Progress Update for April
Good news - the resource issue that has held up progress in March and April has been resolved.
Olly Butters has joined the team and made an immediate difference.
Progress is summarised below:
Data Sources
Olly Butters has joined the team and made an immediate difference.
Progress is summarised below:
Data Sources
The new Cappelli date source is now loaded into the database and being tested.
The 1881 Surname Census data has been reviewed and work to calculate parish centroids is well advanced. Once we have parish centoid data then the information can be added to the database and tested.
Tools for Researchers
The requirements for the 'web based query tool' have been reviewed and agreed with Jayne Carroll of Nottingham University's Institute of Name Studies (the owners of the Key to English Place Names data used by HALOGEN).
Work on developing prototypes has started and the target is to demonstrate the first cut products to the project team and key research users in late May/early June.
The first Project Board meeting is now scheduled for 27th May. Onwards !
Work on developing prototypes has started and the target is to demonstrate the first cut products to the project team and key research users in late May/early June.
The first Project Board meeting is now scheduled for 27th May. Onwards !
Friday, 8 April 2011
Excitement at the GIS tutorial
Last week I gave a GIS tutorial using some prototype data we cleaned up from the original KEPN, PAS and GUL data sets to Turi, Jayne, Phillip, Dave and Mark W. The object was to help empower these blokes by showing them how to load up the data into a GIS environment and chop up the data with some simple querying methods thus stimulating the construction of new research questions.
Talk about the wow factor - they were really chuffed to see a spatial plot of what they used to know as rows and rows of tabulated data. After filtering the data suddenly their hypotheses were mapped out in front of them, e.g. Place names with Cornish elements did gravitate to the county of Cornwall, place names with Norse language elements did gravitate to the North and East of England. When ancillary data such as roads and rivers were plot as background layers I think the cogs and wheels started spinning and ways to answer research questions were suddenly looking so much easier for the researchers.
The data was questioned though, and quite rightly so, it should be a standard procedure for any researcher to be sure of the origins and quality of their data.
(i) Some of the grid references were slipping through our padding procedure and looking too accurate (by this I mean our rounding up of grid references to 0.5km). We did this to ensure privacy of data and maintain a consistent resolution between datasets. This is a small technical issue we need to address.
(ii) Cornish place name elements were detected in Herefordshire and way up in Lancashire. In retrospect Dave and I examined the original data source a few days later and found that these results were true. It was the original data that was throwing up the anomalies, technically the HALOGEN team appeared to get things right.
What do we learn from this? Firstly all the hard work is paying off and the researchers find this a really useful tool. Secondly we can only deal with the data we receive. We did our own quality check to be sure we had it right, if the source data is wrong HALOGEN cannot 'make up' data that fits, a strategy of quality control on the original data is required.
Andrew
Talk about the wow factor - they were really chuffed to see a spatial plot of what they used to know as rows and rows of tabulated data. After filtering the data suddenly their hypotheses were mapped out in front of them, e.g. Place names with Cornish elements did gravitate to the county of Cornwall, place names with Norse language elements did gravitate to the North and East of England. When ancillary data such as roads and rivers were plot as background layers I think the cogs and wheels started spinning and ways to answer research questions were suddenly looking so much easier for the researchers.
The data was questioned though, and quite rightly so, it should be a standard procedure for any researcher to be sure of the origins and quality of their data.
(i) Some of the grid references were slipping through our padding procedure and looking too accurate (by this I mean our rounding up of grid references to 0.5km). We did this to ensure privacy of data and maintain a consistent resolution between datasets. This is a small technical issue we need to address.
(ii) Cornish place name elements were detected in Herefordshire and way up in Lancashire. In retrospect Dave and I examined the original data source a few days later and found that these results were true. It was the original data that was throwing up the anomalies, technically the HALOGEN team appeared to get things right.
What do we learn from this? Firstly all the hard work is paying off and the researchers find this a really useful tool. Secondly we can only deal with the data we receive. We did our own quality check to be sure we had it right, if the source data is wrong HALOGEN cannot 'make up' data that fits, a strategy of quality control on the original data is required.
Andrew
Project Evaluation - ArcGIS Data Exchange Issues
We've hit a problem in exchanging data from the HALOGEN databases with ArcGIS. Up until now we've been exporting the data from the databases into files as tab separated fields, importing into ArcGIS and converting into shapefile data from there. However ArcGIS's shapefile format (or more specifically its associated .dbf database file) has some severe limitations which mean we can't realistically continue down this route.
ESRI has this to say about the use of shapefiles:
With some exceptions that are noted below, shapefiles are acceptable for storing simple feature geometry. However, shapefiles have serious problems with attributes. For example, they cannot store null values, they round up numbers, they have poor support for Unicode character strings, they do not allow field names longer than 10 characters, and they cannot store both a date and time in a field. These are just the main issues. Additionally, they do not support capabilities found in geodatabases such as domains and subtypes. So unless you have very simple attributes and no geodatabase capabilities, do not use shapefiles.
(see Geoprocessing Considerations for Shapefile Output).
Of these limitations, I believe the inability to represent NULL values is the most serious - NULLs are represented as zero within the shapefile's .dbf format, and this means that any value of zero within the data cannot be trusted at all.
I am looking at the ArcGIS Data Interoperability extension so that we can plug ArcGIS directly into the HALOGEN database and avoid these problems. Unfortunately there's no fully-featured solution for connecting ArcGIS into MySQL, so for evaluating this I'll be porting the HALOGEN databases to PostgreSQL.
Liam
ESRI has this to say about the use of shapefiles:
With some exceptions that are noted below, shapefiles are acceptable for storing simple feature geometry. However, shapefiles have serious problems with attributes. For example, they cannot store null values, they round up numbers, they have poor support for Unicode character strings, they do not allow field names longer than 10 characters, and they cannot store both a date and time in a field. These are just the main issues. Additionally, they do not support capabilities found in geodatabases such as domains and subtypes. So unless you have very simple attributes and no geodatabase capabilities, do not use shapefiles.
(see Geoprocessing Considerations for Shapefile Output).
Of these limitations, I believe the inability to represent NULL values is the most serious - NULLs are represented as zero within the shapefile's .dbf format, and this means that any value of zero within the data cannot be trusted at all.
I am looking at the ArcGIS Data Interoperability extension so that we can plug ArcGIS directly into the HALOGEN database and avoid these problems. Unfortunately there's no fully-featured solution for connecting ArcGIS into MySQL, so for evaluating this I'll be porting the HALOGEN databases to PostgreSQL.
Liam
Thursday, 7 April 2011
Project Update for March 2011
In terms of project progress we have achieved less than we planned in March. This is due to one of the key development team members not being available due to a backlog of support work and the need to support another high priority project for researchers at the University (we are developing a hosting service for researchers).
The work on loading new data sources; addressing known problems with ArcGIS and the Key to English Place Names data has all been impacted.
The work on loading new data sources; addressing known problems with ArcGIS and the Key to English Place Names data has all been impacted.
That said the good news is that to address this problem (hopefully for the life of the project !), and to catch up lost time, the opportunity to recruit additional temporary resource has been taken and Olly Butters joined the team on 4th April for 6 months.
Olly has been working on Physics and Astronomy projects and has good MySQL/PHP skills and a lot of experience of curating multiple/large data sets.
The 1881 Surname/Census data has been obtained and is being reviewed.
A discussion paper covering the tools evaluation has been issued of review.
A training workshop has been held for members of the Roots of the British research collaboration to allow them to get to grips with ArcGIS and HALOGEN data.
Requirements for the 'web enquiry' have been agreed and documented with the research user community.
Wednesday, 9 March 2011
"Small WIN(s") & "FAIL(s)" - Progress Update for February 2011
February Update & Issues
The project has now formally started and an internal project proposal has been approved.
Resources have been allocated and the contract of our GIS specilist has been extended to cover the specialist GIS resource required to support this project.
Good progress has been made on the Data Sources Work Package (WP2). The HALOGEN system has been extended to include a full down load of Portable Antiquities Data (PAS) for all English counties. The increase in the volume of PAS data has created problems with the use of the ArcGIS tool used by the team (see issue below).
The project team have changed the scope of the Data Sources work slightly and agreed that 2 new sources of data will be added. These will be additional Genetics data from the published datasets of Capelli and the 1881 Surname/Census data from Kevin Schurer’s work with the Essex Data Archive at the University of Essex.
This later source is a replacement for the surname distribution data from the Archer Surname Atlas originally referenced in the project proposal. Initial investigations identified issues with the use of this data which have been addressed by the Essex Data Archive.
Requirements relating to the Capelli data have been agreed and the source data obtained. The data has been enhanced by the addition of relevant spatial data. Design work to include this in the HALOGEN database is in progress.
An initial meeting has been held to review the format and content of the surname census data from the Essex Data Archive and further sessions are planned.
Extension of PAS Data – Impact on ArcGIS
The addition of extra PAS data has caused problems with the use of shape files in ArcGIS. To temporarily overcome this issue PAS data is being extracted in 3 separate files for use with ArcGIS. The team continue to investigate a permanent fix to the problem.
Any hints or tips welcome !
The project has now formally started and an internal project proposal has been approved.
Resources have been allocated and the contract of our GIS specilist has been extended to cover the specialist GIS resource required to support this project.
Good progress has been made on the Data Sources Work Package (WP2). The HALOGEN system has been extended to include a full down load of Portable Antiquities Data (PAS) for all English counties. The increase in the volume of PAS data has created problems with the use of the ArcGIS tool used by the team (see issue below).
The project team have changed the scope of the Data Sources work slightly and agreed that 2 new sources of data will be added. These will be additional Genetics data from the published datasets of Capelli and the 1881 Surname/Census data from Kevin Schurer’s work with the Essex Data Archive at the University of Essex.
This later source is a replacement for the surname distribution data from the Archer Surname Atlas originally referenced in the project proposal. Initial investigations identified issues with the use of this data which have been addressed by the Essex Data Archive.
Requirements relating to the Capelli data have been agreed and the source data obtained. The data has been enhanced by the addition of relevant spatial data. Design work to include this in the HALOGEN database is in progress.
An initial meeting has been held to review the format and content of the surname census data from the Essex Data Archive and further sessions are planned.
Extension of PAS Data – Impact on ArcGIS
The addition of extra PAS data has caused problems with the use of shape files in ArcGIS. To temporarily overcome this issue PAS data is being extracted in 3 separate files for use with ArcGIS. The team continue to investigate a permanent fix to the problem.
Any hints or tips welcome !
Project Plan Post 2 of 7: Wider Benefits to Sector & Achievements for Host Institution
The benefits from specific project deliverables are listed in the table below.
Deliverables | Benefits and Outcomes |
1. Enhanced Data Sources – Extension of existing PAS datasets to provide national coverage. This is a relatively simple exercise as the Key to English Place Names and Genetics data is already present in the database at national level but some additional cleaning is required. A new ‘national’ extract of PAS data would be needed. - Addition of 2 new data sources relating to the geographical distribution of surnames and further genetics data to the database. | A review of existing HALOGEN data extraction, cleaning and load procedures to cope with ingestion of ‘national’/larger data sources. A prioritised list of data source related requirements that can be used to guide the future development of the service. Data extraction, cleaning and load procedures for new datasets. Updated data glossary for researchers. |
2. A Revised Data Management Plan Additional requirements, policies and practice recommendations covering new features will be documented. | An assessment of the effectiveness of the DCC’s drafts DMP as an aid to research data management. Information and lessons learned to source a JISC case study and input to community synthesis project. |
3. Evaluation and Selection of Data Extraction Tool Requirements for an appropriate tool will be documented. These will initially be used to assess the feasibility of using the existing tools supported by IT Services. If this is not appropriate a market evaluation will take place. | A contribution to wider JISC community to help develop awareness of good practice in terms of the selection and availability of similar tools. |
4. An Implementation Plan for the Data Extraction Tool An implementation plan covering the timescales, costs, risks and issues relating to the deployment of the selected tools will be documented. If feasible within the 9-month project window then the preferred product will be implemented. | |
5. Feasibility Study for the Provision of HALOGEN Database Enquiry Facilities for Institute of Place Names Website Users. | A contribution to wider JISC community to help develop awareness of good practice in how to deal with similar requirements and problems. |
6. Interim project reports | Compliance with JISC requirements for project control. |
7. A project blog and updated Halogen Project website | Sector-wide dissemination of findings and engagement with key stakeholder communities. |
Project Plan Post 7 of 7: Budget
Budget Summary
The total cost of the project was c.£353,000 of which the JISC award of £85,000 represents 24% of the total project costs. A summary of the total project budget (covering both JISC and institutional contributions) is given below.
Category %
Directly Incurred Staff 5
Directly Incurred Other 7
Directly Allocated 50
Indirect 38
Total 100
The largest forecasted costs relate to staffing. It is estimated that a 'virtual team' of 3.27 FTE's will work on the project for it's duration.
Budget Management
The project manager will be responsible for managing and monitoring the project budget on a day to day basis. The project manager will be accountable to the Project Board and will report any significant variances to the Board for discussion and authorisation.
The project board is chaired by Professor Annette Cashmore (Sub-Dean for Medicine and Biological Sciences, Director of CETL (GENIE)) and its membership includes David Flanders, JISC Programme Manager, Professor Mark Jobling (Department of Genetics), Dr Jayne Carroll (Director Institute for Name Studies, University of Nottingham), Mary Visser (Director of IT Services) and Dr Nick Tate (Senior Lecturer, Department of Geography).
The total cost of the project was c.£353,000 of which the JISC award of £85,000 represents 24% of the total project costs. A summary of the total project budget (covering both JISC and institutional contributions) is given below.
Category %
Directly Incurred Staff 5
Directly Incurred Other 7
Directly Allocated 50
Indirect 38
Total 100
The largest forecasted costs relate to staffing. It is estimated that a 'virtual team' of 3.27 FTE's will work on the project for it's duration.
Budget Management
The project manager will be responsible for managing and monitoring the project budget on a day to day basis. The project manager will be accountable to the Project Board and will report any significant variances to the Board for discussion and authorisation.
The project board is chaired by Professor Annette Cashmore (Sub-Dean for Medicine and Biological Sciences, Director of CETL (GENIE)) and its membership includes David Flanders, JISC Programme Manager, Professor Mark Jobling (Department of Genetics), Dr Jayne Carroll (Director Institute for Name Studies, University of Nottingham), Mary Visser (Director of IT Services) and Dr Nick Tate (Senior Lecturer, Department of Geography).
Project Plan Post 6 of 7: Projected Timeline, Workplan & Overall Project Methodology
Work Plan The high level workplan is outlined below. Listed against each workpackage are the initials of the principal team member(s) responsible for its delivery.
Workpackage ò | 02 11 | 03 11 | 04 11 | 05 11 | 06 11 | 07 11 | 08 11 | 09 11 | 10 11 | ||
WP1 - Set-up and governance (DC) | |||||||||||
Project set-up, induction and PID | |||||||||||
Steering and project group meetings | |||||||||||
JISC Programme level activity and reporting to funder | |||||||||||
WP2 – Data sources (AB, OB, LG) | |||||||||||
Extend coverage of PAS data | |||||||||||
Additional data sources - establish detailed requirements and investigate feasibility for 2 new sources. | |||||||||||
Extract, clean, transform and load new data sources | |||||||||||
Update data glossary and system documentation | |||||||||||
WP3 – Update HALOGEN data management plan (DC) | |||||||||||
Review and update DMP | |||||||||||
WP4 – Data Extraction Tool (AG, MW) | |||||||||||
Establish and document requirements | |||||||||||
Evaluate tools and select preferred supplier | |||||||||||
Procure and produce implementation plan for preferred tool/supplier | |||||||||||
WP5 – Web Enquiry Facility (OB) | |||||||||||
Establish requirements | |||||||||||
Investigate options and feasibility of delivery | |||||||||||
Document and publish findings | |||||||||||
WP6 – Develop & Disseminate HALOGEN Case Study (DC, OB, AB) | |||||||||||
Prepare case study re:community syntheses input | |||||||||||
Disseminate to stakeholders | |||||||||||
WP7 - Project Evaluation (DC) | |||||||||||
Develop interim and final project reports (see evaluation milestones below) | |||||||||||
WP8 – Dissemination (DC, OB, AB) | |||||||||||
Maintain project blog, halogen web site and run briefing sessions | |||||||||||
Project Approach
When identifying and evaluating specific tools and technologies, open source solutions will be considered alongside those which are already licensed by the University, the aim being to reduce the ‘barrier to entry’ for other institutions wishing to adopt the approaches used by UoL.
The evaluation will involve some desk based research but will be heavily biased towards the building of 'prototypes' using different tools.
Subscribe to:
Posts (Atom)