Yale Library Quicksearch Project News
Quicksearch Ingest Activity for February
Orbis:
83,928 records updated or added
4,334 records deleted
52 ingest errors
Morris:
9,583 records updated or added
784 records deleted
0 ingest errors
Changes to records from Orbis and Morris are transferred to Quicksearch daily. Errors preventing ingest are usually an empty subfield or empty delimiter.
Quicksearch Ingest Activity for January
Orbis:
64,436 records updated or added
9,403 records deleted
44 ingest errors
Morris:
4,786 records updated or added
154 records deleted
1 ingest error
Changes to records from Orbis and Morris are transferred to Quicksearch daily. Errors preventing ingest are usually an empty subfield or empty delimiter.
Quicksearch January Update
Dear Colleagues,
I’m happy to announce that Quicksearch will become the default search on the Library Front Door on Tuesday, January 12th!
What does this mean?
As of January 12th, the library front door will have a new design, courtesy of the User Experience Advisory Group. You can see a preview of the new design at http://dev.library.yale.edu. Although Quicksearch will be the default search on the Library Front Door, links to Orbis, Morris, and Articles+ will still be prominent on the new page.
More details about the latest Quicksearch developments…
Members of the Quicksearch Implementation team and many other library staff worked hard in November and December to get Quicksearch ready for this day, and I’d like to highlight some of their efforts.
Requests – Orbis, Morris, Scan and Deliver, and Aeon request links have been added to Quicksearch! The request links themselves have been added in the holdings area, next to the locations they are associated with. Thanks to Yue Ji, Lakeisha Robinson, Tracy MacMath, and Scott Matheson who helped develop the request links and the new holdings area layout. Thanks also to all the staff across the library system who participated in testing the new request links, and who gave feedback about their placement.
Books+ Search Analysis group documentation – the Books+ Search Analysis group, led by Arcadia Falcone, finished posting their documentation to the Cataloging at Yale web site. You can see a detailed description of Quicksearch indexing at http://web.library.yale.edu/cataloging, under Discovery Systems Documentation. The documentation describes both indexing changes made this fall at the group’s recommendation, and those that are still under consideration for future implementation. Many thanks to everyone on the group for their hard work.
SearchFest – our final SearchFest was held November 17th. This was the fifth and final Quicksearch SearchFest – many thanks to Jenn Nolte for leading these pizza and fun-filled learning opportunities, and thanks to all the staff who participated and presented their own tips and tricks for searching in Quicksearch!!
Infrastructure – we are working closely with ITS to identify issues related to occasional 5-10 minute outages of the alias search.library.yale.edu. Quicksearch itself is still up, as is the ITS load balancer, but the alias is locked. Thanks to Bob Rice for taking the lead on this issue. On the bright side, Quicksearch server performance testing concluded successfully in late November with our individual servers handling a load at 20% above the maximum number of simultaneous users experienced historically by Orbis during the busiest time of the year.
User Experience Advisory Group – thanks to this group led by Steven Wieda, who, in addition to creating the new layout on the Library Front Door, also reviewed the layout and appearance of the new additions and features in Quicksearch this fall.
Finally, thanks to all of the Library staff who have participated by providing feedback this year. The project wouldn’t be where it is without all of you!
Next up – this is not the end of the Quicksearch project, merely another important stage. As we move through the winter and spring months, Quicksearch will continue to address outstanding issues and add new features, including the addition of a new digital materials bento box from the Findit digital interface.
Quicksearch Ingest Activity for December
Orbis:
89, 057 records updated or added
10,138 records deleted
61 ingest errors
Morris:
3,375 records updated or added
134 records deleted
0 ingest errors
Changes to records from Orbis and Morris are transferred to Quicksearch daily. Errors preventing ingest are usually an empty subfield or empty delimiter.
New documentation available for librarians and library staff
New documentation for Quicksearch is now available on the Cataloging @ Yale site, under Discovery Systems Documentation. This documentation is aimed at librarians and staff, and answers questions such as:
When I search “Author”, what MARC fields am I searching? (Quicksearch Search Indexes)
Where do the facet terms come from? (Quicksearch Facet Indexes)
Why are all the RDA records at the end when I sort results by publication date? (Quicksearch Sort Indexes)
How does Quicksearch decide which format terms to assign to a resource? (Quicksearch MARC Format Mapping)
I made a change in Orbis…when can I expect to see it in Quicksearch? And how can I make sure my record won’t fail to ingest into Quicksearch? (How and When Cataloging Data Appears in Quicksearch)
Why aren’t all my search terms in some of my results? And what special syntax can I use to improve my searches? (Quicksearch Expert Search Tips)
What really happens when I click the search button in Books+? And how does the Quicksearch relevancy ranking work? (Technical Overview of Searching in Quicksearch and Books+)
How can I keep up to date with Quicksearch developments? (Quicksearch Project Blog)
Where do I point patrons for more information? (Public Quicksearch Help Site) (work in progress)
Many thanks to Dominique Bourassa for transforming these documents into Drupal, the members of the Books+ Search Analysis Working Group for their work on documenting the Quicksearch Solr indexes, and attendees of SearchFest for informing this documentation through their presentations and questions!
Quicksearch Update November 2015
The Quicksearch implementation team has been busy this fall with two major projects.
The first project was to improve the indexing of Morris and Orbis records. Work began with the Books+ Search Analysis Working Group, a broad-based team of technical services stakeholders, led by Quicksearch team member Arcadia Falcone. The group wrote a recommendation document highlighting changes to searching, faceting, and sorting in Quicksearch. The changes were then reviewed by the Cataloging Coordinating Committee and the User Experience Advisory Committee. The approved recommendations were then implemented by Quicksearch team members in a series of code updates and a full extract and re-load of over 10.3 million Morris and Orbis records. A new Subject (Local Yale) facet is just one of the recommended changes now available in Quicksearch. A few recommended changes related to facet and sort labels are still in progress and will be available in production Quicksearch within the next week.
This fall the team has also been busy improving the layout, design, and content of holdings in the Quicksearch single item view. A new streamlined design was introduced to Quicksearch earlier in the fall, with improved indentation designed to make it easier to see individual holdings for items with numerous holdings statements. Holdings notes relating to the Provenance and description of individual special collections items were also added to the holdings display. Finally, the mobile view of holdings has been revised, and is continuing to undergo review.
Currently on the development front – the team has now embarked on the project to implement Aeon and Scan and Deliver requests in Quicksearch. In addition to adding new request types, we will re-design the request links in the item view, making the options to request from Orbis, Morris, Scan and Deliver or Aeon, easier to find and use for our patrons. We hope to make all of these request types available in time for our January roll-out of Quicksearch.
Quicksearch full reload statistics
During the last week of October, Quicksearch was refreshed with a complete reload of records from Orbis and Morris, and a new indexing configuration was implemented. Following the reload, a complete audit of the Solr index identified any records that were not correctly transferred to or deleted from Quicksearch.
Orbis:
9,843,283 records extracted
9,841,924 records initially loaded
343 ingest errors for correction (a significant drop since the last reload, thanks to the ongoing efforts of CMS and other technical services staff)
Remaining difference re-extracted
Morris:
499,410 records extracted
498,941 records initially loaded
Remaining difference re-extracted
Quicksearch Ingest Activity for October
Data is through October 26th, after which the daily ingest was put on hold for a full reload of records into Quicksearch from Orbis and Morris.
Orbis:
122,633 records updated or added
1,348 records deleted
68 ingest errors
Morris:
8,370 records updated or added
156 records deleted
106 ingest errors
Changes to records from Orbis and Morris are transferred to Quicksearch daily. Errors preventing ingest are usually an empty subfield or empty delimiter.
Quicksearch Ingest Activity for September
Orbis:
79, 164 records updated or added
6,633 records deleted
108 ingest errors
Morris:
8,136 records updated or added
187 records deleted
4 ingest errors
Changes to records from Orbis and Morris are transferred to Quicksearch daily. Errors preventing ingest are usually an empty subfield or empty delimiter.
Quicksearch ingest activity for July and August
Now that the daily Quicksearch ingest audit has been up and running for several weeks, we have data to share! These statistics demonstrate not only the volume of records going into Quicksearch, but also the work that Yale and Law Library staff put into creating, editing, and managing records in Orbis and Morris. The count of records added to Quicksearch includes all records that were created, loaded, updated, or unsuppressed during this time period.
And now, the numbers:
From Orbis, 99,271 records were added to Quicksearch in July, and 89,486 in August. July saw 357 records failing ingest due to MARC errors (usually an empty subfield or rogue delimiter lurking somewhere), 77 of which were corrected by Catalog & Metadata Services staff. The remainder are preliminary order records that will be updated when the item is received, or records that triggered an error on multiple occasions. August’s error count was 107, of which 40 were corrected. The deletions for July were 2,540, with 1,779 records deleted in August.
From Morris, July’s additions numbered 6,944, with 224 deletions. The August counts were 17,829 records added, and 111 deleted. For both months Law maintained a clean slate with no errors logged.
Grand totals: 106,215 new or updated records in Quicksearch in July, and 107,315 more in August.
ETA: These monthly statistics are derived from a daily audit of Quicksearch loads, so if the same record is updated in Orbis or Morris several times over multiple days, each day on which it has new changes triggers re-ingest into Quicksearch and adds one to the tally.
Morris Holdings now available in Quicksearch!
Morris holdings now appear in Quicksearch!
The best way to view this new feature is to search for a law-related topic like ‘Human Rights Law’ in Quicksearch: http://search.library.yale.edu.
In the Search Results list you can now see real-time availability information. A red ‘x’ appears if the title is checked out, and a green check mark displays if the title is available.
In the individual holdings screen, you can see the location, call number, and status of the title pulled directly in real time from Morris.
This is an important milestone in the Quicksearch project ; the two Yale Library catalogs, Orbis and Morris, are both now fully represented in Quicksearch.
New Audit Tool Keeps Quicksearch in Sync with Library Catalogs
While most MARC records transfer smoothly from Orbis and Morris to Quicksearch, sometimes things go wrong and discrepancies between the systems arise. The recent development and implementation of an automated daily audit process ensures that these errors will be identified and routed for correction swiftly.
The audit script runs each time Quicksearch processes a fresh extract of records from Orbis or Morris. The script identifies the record IDs from the processed extract files. It then contacts the Quicksearch Solr index to acquire an up-to-date list of all the record IDs in Quicksearch, as well as the date each record was last updated. By comparing the extract files to the contents of the index, the script checks that all added records are present with a current update date, and that all deleted or suppressed records have been removed. A notification e-mail immediately reports the audit results to the Quicksearch project team, and includes file attachments listing the record IDs that failed to be processed correctly. These errors are collected, analyzed to determine appropriate handling, and routed to Catalog & Metadata Services or Law Library staff for review.
The next phase in development is for the script to send the daily audit results to a local database, which will in turn power a publicly available visualization that updates in real time, illustrating the volume of catalog records that YUL staff create or update on a daily basis, and charting their flow from Orbis and Morris into Quicksearch.
Quicksearch Update June 2015
The Quicksearch project team has been working hard in the past few months to implement several important “behind the scenes” improvements, and I’d like to highlight a few of them for you:
- Production Quicksearch is now supported by not one, but two separate servers that are mediated by something called an F5 load balancer. An F5 load balancer acts like a traffic light; for each incoming user request, and it will look to see which of our two production servers is least busy, and direct traffic to that server. This balanced approach has several advantages. First, Quicksearch will be able to handle the level of traffic currently experienced by Orbis and Orbexpress. Second, in the event of a server failure, the load balancer will automatically direct all traffic to the server that is currently up, with no visible downtime for users.
- We are implementing a new approach to Google Analytics that will enable analytics-driven change in Quicksearch. Quicksearch has been upgraded to use both Google Analytics Universal code and the new Google Tag Manager application. Google Tag Manager will allow us to more easily track user behavior. We also developed custom code specifically for identifying the broad use of items by location and call number. The setup was driven by small group discussions between Library IT and Sarah Tudesco, with the goal of setting up analytics using similar approaches in all of the applications supported by Library IT. These new analytics will be analyzed over the summer and tweaked as necessary. Our goal is to finalize our setup by September 2015.
- Daily record extract and load workflows are being finalized. For example, a daily Audit script of all loading activities has been added. The Audit script generates a list of records rejected by Quicksearch for clean up in Orbis or Morris. The statistics associated with each load will be publicly available in the near future in a graphical dashboard. A full audit workflow also now exists to compare all publicly-available records in Orbis, Morris, and Quicksearch, and generate a list of any records out of sync between the three public OPACs.
What’s up next in the Quicksearch project?
- Work is in progress to integrate the display of Morris holdings.
- We’re also creating public-facing Help documentation for Quicksearch users. A sub-group of the Quicksearch Implementation team has formed to create this documentation, and will be working in consultation with the User Experience Strategy Group.
- Re-discover Discovery and other staff sessions will be planned for late summer to support library instruction, in consultation with public services staff.
- Quicksearch is still on target to replace Orbis as the default search on the Library Front Door in January 2016!