Now that many VTNLISTS are installed, it is time to use them.   For those titles that we decide NOT to set up with its own order, I will precede the Title Number with an “X” and (if I remember) program the algorithms to ignore (do not report) these as problems when detected.

In other words, the program will take a title number from the SOP and look for it in Voyager.   If it is not found in the usual spot (on the VTN location under the “TYPE” tab on a Line Item), it will then turn to the VTNLISTs.    If it sees one there that is preceded with an X it will stop its analysis and go to the next item in the SOP.   If it sees on there that is NOT preceded with an X, OR if it does not see the title number, it will report as a problem.

Just sayin’!

eof

Qwerty:   1. Made up of characters that one may type using a standard QWERTY keyboard.  2. Characters that are made up of the ASCII set of printable characters numbering from 32 to 126 that represent language/expression characters (this excludes machine commands such as Delete, Backspace, Escape etc.).

I may elaborate on this definition later, but generally the above definition works for the purposes of this discussion.

Okay.

Well, this Unicode thing is a bit tougher than expected.     I’ve been able to account for and convert all of the Unicode (or special characters) within our title list (from our bib records), but the SFX/Aggregator titles are a whole different matter.

For one, some have no Western or Arabic characters at all.    That is, they are completely Chinese, Thai, Vietnamese, etc., with no alt titles.    This means that there cannot be any equivalent or even close character substitutions.   That only leaves the previous method of transliteration: using the old “agg_weird.dat” file that I’ve built over time.

This is frustrating because these titles do not display as Qwerty titles.     Some of these titles do not even have ISSN or ISBN numbers!     This means I must do some research each time one comes up and either copy or invent some Qwerty version of the title.

In short, 100% conversion of every title that may possibly exist (now or in the future) is not possible using any approach that is now within my ability or resources (including time).

So-o-o,  I’ve decided to develop a “hybrid” conversion process.

Here’s how it will work (I hope).

Journal List Updater programs encounter a special character within a string.

• First, an attempt is made to “normalize” to a Qwerty form using a series of algorithms.    If this works, all is well and the program proceeds with its other tasks, skipping the below steps.

• Second, if the above fails, the program will further examine the string to determine if the existence of the special character might cause a break in the sequence of data (big danger!).
•               >>> If a potential data break (such as a line break) IS detected, the program will SKIP that record, RECORD the suspect data in a separate file to safely isolate it from other records, then SEND a notice to the Administrator (me) that a problem was encountered.

• Third, if a potential data break IS NOT detected, the “agg_weird.dat” file is consulted using a key from the problem data (such as an ISSN, ISBN, or Object_ID number).
•             >>>If a match IS found, a substitution will take place based on the record within the file.     This is more or less a “dictionary” look up.
•            >>>If a match IS NOT found, the entry will be recorded as a problem and a notice about it will be sent to the Administrator.

I think that, while inelegant, this scheme will work.    It’s the one I’ll be going with, anyway.    At least this method will permit the data processing programs to proceed (right now, everything halts until I can intervene) without “supervision.”      So if this works, I’ll be satisfied.

I suppose.

eof

I want to standardize journal titles for several reasons:

• To make them legible
• To make them searchable using a standard keyboard
• To prevent them from “breaking” data with field marks
• To enable the Journal List data prep programs to run with intervention (right now, when they hit a non-standard character, they pause, waiting from me to make some decision before proceeding)

So I downloaded all of the titles in our online catalog data (not just journals) knowing I would find problems.    I was thinking the list would serve as a good set of data to troubleshoot from.

I was right.

In case you’re reading this and wondering what the big deal is, consider this:   There is not only the Standard ASCII, but also the Extended ASCII set of characters.    Since Unicode first came along, there has been at least 8 changes in what is “standard” Unicode.    In addition, just in the “current” standard alone, there are 108 Unicode “sets” of characters (see Unicode Tables) !!!    And each of these tables may have hundreds of individual characters.

Now, in the good ol’ days, everything was “normalized” into ASCII.   Then Extended ASCII came along.   255 characters.   Plenty!   Also, English was the standard for programming in English-speaking countries.     American English in the United States.

In my work, it is apparent that there has been no institutional will to stick to any standard when it comes to record sets and characters used.    Hence, our problem.

Our records date back to the late 1970′s when we first started automating.    Since then, every imaginable error has been made.   Incorrect data entry.    Poorly standardized “purchased” records.   Overlays of “new” records over old records, even if they do not conform to the previous record’s standards.    Marc formatting codes and symbols accidentally (or maybe intentionally… who knows?) used as characters in titles.   Non-English character sets imported and used that cannot be typed on a standard QWERTY keyboard (even with ctrl/alt/shift combinations).

This leaves us with a veritable “anything goes” soup of possible problem characters that may crop up at any moment to kill, maim, or mutate not only the record that it exists within, but other records around it.

THAT’s my problem.     The challenge is to find them and “normalize” them.    Since no “normalization standard” exists, I’m trying to come up with one that will work for MY applications.    And, sorry world, ASCII is MY standard.    It is compatible with any machine that my applications will be used on.    So there.

So, I’m running an analysis program that spits out all non-ASCII anomalies for me to make a decision about.    Not only am I finding the expected non-English characters, but I’m also finding the most ridiculous “Unicode” characters used in place of standard intelligible characters.     Why, for instance, would you put a Unicode dash (ORD 226 128 147 or ORD 226 129 187) when you could use a good ol’ ASCII dash (chr 45), especially when all of the rest of the characters are ASCII?.    ‘Cause, guess what?   That Unicode dash is not interpreted as such by many data processing programs and machines.    Just an example.

So, even though my programming is saving a lot of time, it is very tedious.   My character “replacement” choices are imperfect.   Some are obviously wrong (in terms of the base language), but I’m trying to stick to characters that EXIST on a standard keyboard!!

I’ve been at this for over a week  with very few breaks.

When I wrap this project up, I hope I’ll be able to use the algorithms in this program as some sort of subroutine for checking entries in the Journal List.    Right now, the Journal List prep program stops when it encounters a non-standard character in a title.      It then waits for me to add it to a reference file before proceeding.    So obviously, I’d like for the Journal List prep program to run independent of my intervention, hence all this work.

I will also try to report back how many titles are in our online catalog that cannot be found with a standard keyboard search.

Until then…

eof

More study is needed before I will be able to come to a conclusion.

Meanwhile, my users are very frustrated.    And so am I!!!

eof

So I’m installing these lists on regular packages where there’s room.   I’m also renaming the “PROXYLIST” to the more accurate “VTNLIST” (meaning, Vendor Title Number List).    For the nearly 420 that PROXYLISTs that I’ve already installed, I’ll go back and recode them using a macro.

For lists that are too long, I’ll install this:    “VTNLIST:(external)”

This should clearly indicate a long list exists elsewhere.   Where?   That is yet to be determined, but may be located in special directories, in License Abstracts, or at some other location.   Perhaps, as things evolve, a more specific Resource Location, or even a URL may be contained within the code.

BTW, if you are wondering by now how these will be actually used during reconciliation, stay tuned.    I’ll be detailing that in a upcoming blog entry.

eof

That’s just one example of many situations that we may address using the Package Proxy approach.    It is an easy fix for otherwise snarly problems.

Membership Proxies are more straight-forward, actually.    They are vended to us as a package of stuff provided through a membership.    Often, these packages are comprised of publications produced by a society or an association of some kind.    We pay for a membership and get them all.     Sometimes, we have no choice; these organizations are not sophisticated enough to handle subscriptions outside of their membership models.     And that’s how come an institution can be a member of a club!

Yes, memberships are a sub-type of package.     A membership or package “proxy” is simply when we do not have a specific order set aside to represent the membership itself, or the package at large.    Instead, we rely on a regular order for one of the titles provided through the membership or package to act as the “central entity” which unifies the rest of the titles/orders together, to act as the proxy for the “absent” package or membership order.

And, yes, we can and may, at any time, create a new order to represent the package or membership and to take over the roles formerly associated with the proxy order.     This is always desirable, but not always convenient.   Sometimes we are forced to do this when the title we selected years ago to be the proxy ceases publication (yes, it happens!!).

BTW, we currently have 278 individual Proxies.   Combined they link together over 769 titles (based on a count of our new PROXYLIST entries).

eof

; PROXYLIST(472011303 472010248 472011030);

These are experimental entries on around 111 orders, so far.

Let me offer a somewhat simplified explanation:

We have many packages for which we have not yet created an order to act solely as the “Package Order.”   Instead, we have selected one of the titles of the package to use as the “Package Proxy,” and, using that PO, we can unite all of the members of the package together.

However, there are some problems with using the “Package Proxy” method.   For one, EBSCO may designate a title entry as the “Combination / Membership / Package” title and attach our PO number to it.   Or, EBSCO might include other title entries to represent supplements, memberships, or additional titles.   In either case, when we attempt to reconcile our records against their records, something will always be missed, unless we’ve set up an order for each and every item in EBSCO’s list of what comes on a package.   Not good, since we don’t actually check-in or keep some things that “come with” these packages.

So the PROXYLIST, contains a list of every EBSCO title number that is associated with an EBSCO “Combination / Membership / Package.”   Or, it may represent a list of EBSCO title numbers that we associate with an order.   Either way, it gives us a method of accounting for stuff.   Our reconciliation programming can parse out the title numbers and use them in association with other data when analyzing the Summary of Publications data.  Hopefully, this will result in fewer problems being reported.

Why don’t we use this for all packages?   It would be impractical.   For one, “Proxy Packages” tend to have only a few titles.    Regular packages may have thousands and are accounted for using a different method, so the PROXYLIST is not needed.   And, anyway, package proxies are on their way out; the tendency is to create a controlling purchase order for these, just like “real” packages, so that as titles come and go, we can account for them.

I will using this same method for “Membership Proxies” (we have about 170 of those), so that we can hopefully account for those titles, too.

I hope to run the reconciliation routines again in a week or so.    At that time, I’ll determine if these PROXYLISTs actually work.

eof

But I have been working.    I’ve made many changes to the EAVS, including the beginnings of a Problem Resolution Module.   I changed how the Journal List handles canceled orders, and I’ve made updates to the Unified Newspaper Database (data).    I’ve tweaked the License Abstracts interface, too, to permit a new “Content Access Begin” note field (intended for some general date info).   I’ve also been working on a way to do and “order reconciliation” with EBSCO, mostly building up to it by trying to locate and install the correct Ebsco Title Numbers into our Voyager data.

If I get a chance I’ll elaborate on these activities.

When I get a chance!

eof

In the past, I’ve tried to incorporate new data mining/analysis routines into the Updater as subroutines.       That might still be the best way to go, long run.   But I do want to create a batching process where certain of these other programs are automatically run as needed.   The idea is that one batch program would launch a “controlling” program and it would determine which of the “suite” of programs available to it should be run, skipping unnecessary ones (ones that show no new data for them to use).

I’m not quite ready to do this, but I am seriously looking at it as a way to make the Journal List updating process automatic and, hopefully, more frequent.

In the longer term, I’d like to migrate much of the Weekly Data Run routines to the Journal List server, and start controlling that batch in the same way.   Like the EAVS, I’d like for the Journal List to also drive POLARIS, Claims Wizard, etc.     Then, I’d like for updates to run nightly.

eof

Sometimes backlogs can happen due to all the vicissitudes of bureaucracies, including staff changes, system changes, etc.   And “to do” lists can be lost or covered up by other work.

So I am asked to redo these lists from time to time.    No biggie.   However,  since I don’t do this work regularly or frequently, I get rusty.    I have to relearn the process, dig up or rewrite the programming, etc.

That’s why I’m creating a method of doing this mining and trying to work it in as a regular part of the Journal List update process.    The routines, which exist as separate programs outside of the Journal List updating program, produce data results lists.   What I’ve done is modify the Journal List updating program to look for this list and to code records within the Journal List with a “needcat” code.   That way, they can be easily located by Journal List users.   At the same time, it also produces an Excel file of these records that can be downloaded and used as a worklist.

Right now, until I become really confident in the NeedCat programming, I launch the routines manually.   Eventually, they will become part of the regular batch of Journal List updating programs and will automatically run.     See my other blog entry about the Journal List Updating Suite.

eof