The fact is that all old newspapers ever published in the United States are not online. Much less than you might think. I recently did a webinar about using Chronicling America to its fullest and obtaining more articles. In so doing, it was an interesting comparison of the number of online newspapers on that site versus the number of newspapers ever published in America. What is your guess of the percentage? 10%? 20%? 5%? No, the answer (at least with respect to Chronicling America) is that there are approximately 3,400 online titles and there have been over 156,700 titles ever published in the U.S. That yields just a tad over 2.17%. That's less than 3% ! Of course, Chronicling America doesn't represent all of the online titles and digitized pages available. My research results in about 35,000 free online links for the U.S. (see the Newspaper Links page on this website for all the links). So 30,000 (online) in comparison to 156,000 (published) - that's about 19% of the titles. What about some of the paid sites, such as GenealogyBank, Newspaper.com, etc.? GenealogyBank on their website claims over 13,000 titles and Newspapers dot com states 18,000. And that doesn't count My Heritage or FindMyPast's collections. And there is considerable overlap probably. So if we are being generous, let's add another 15% for the paid sites. That says that 30% have been digitized. That's titles, NOT pages. A single title that has one page from one issue in our math example counts the same as a title published daily for every year for a hundred years. The moral of the story is that with all the date gaps that we newspaper researchers run into - it is almost impossible to figure out how much indeed has been digitized. So let's just say for now that 15 to 30% of all U.S. newspaper pages have been digitized. Personally I think that's a very high number. That leaves about 70-85% that are NOT online! Now what the heck do we do? Offline research - that's what. And the Library of Congress helps us out with the US Newspaper Directory from 1690 - Present so we can do offline research. This handy directory provides not just multiple ways to find newspapers published, but tons of facts about those newspapers. Just do your choice of searches from the main US Newspaper Directory from 1690 - Present page. Here's an example of a "fact" page". It is presented in two parts. And here is the bottom "half" of that display: See all the interesting information about the paper? If you had an ancestor from the area that you think might be mentioned in this paper - and the paper was not online - maybe you could find it at an archive or library in original form or microfilm. From this "facts" page, you can actually find out where the newspaper is held. Just click on the "View complete holdings information" link at the bottom. Here are some of the results for this newspaper: So, you can see that it is available for your browsing pleasure at several libraries and archives - both in original and in microfilm form.
Don't give up if your paper of interest is not online. Visit the US Newspaper Directory from 1690 - Present from the Library of Congress.
0 Comments
I love searching old newspapers and have become addicted to searching them. And the majority of this website is dedicated to helping you get the most out of newspaper research. What things can one find in these sometimes one hundred or hundred and fifty year old pieces of paper? Here's a list of items that you may or may not have considered:
So - check out old newspapers. See what you can find. You might be amazed! Everyone who has searched newspapers online will fail to find something. It happens incredibly often. The stakes are high for genealogy researchers, where finding an article about an ancestor can make a huge difference in filling out a family tree or for historians, where one can find an extraordinary story unknown before. I have often heard researchers say "I can't find a single article about my subject, even though I have searched for hours!" Laying competency aside as a factor, the biggest reason is that the scanning and OCR of one and two hundred year old newspapers, either from paper or from microfilm, produces way less than optimal results. Here are 5 secrets to success: 1. Understand that you are trying to search against an imperfect database Every collection of digitized newspapers has two parts. The first is the collection of all the scanned images. The second is the OCR text that contains all the "words" in the newspapers. In fact, the OCR text does NOT contain words or names - it really is just a string of letters and spaces. You need to know AND remember that the key is that it is a string of letters, not NECESSARILY a word or a person's name that you are searching against. The OCR process is intended to convert the dots on the page to letters, numbers and symbols (such as punctuation). If the original newspaper is dirty, brittle, creased, etc., the OCR process, which is applied to the scanned image, may not convert the dots to the correct letters. There are a variety of strange characters and combinations of strange characters that may be picked up. The bottom line is that the OCR text - a representation of the dots from the original pre-scanned newspaper page is only as good as the quality of the original, the quality of the scanner, and the quality of the OCR software. That's a lot of "quality" that has to exist. You must expect inferior results and set your expectations accordingly. 2. Dates and Location Matter A digitized newspaper collection represents a number (sometimes a large number) of newspaper pages that when combined are a newspaper edition for a specific day (or part of a day if there is a morning and evening edition). So, a page is part of a newspaper title's publication for that day. Several days and multiple titles become a collection. Let's say that the digitized collection of the "Herald News" has publication dates of November 12, 1887 through January 31, 1894, which represents all the newspapers in the collection. No matter how hard you try you cannot make the collection give you results for your great grandfather's obituary if it was published in the Herald News on March 14, 1896. A very common mistake that online newspaper researchers make is to ignore the dates of newspapers that are in the collection and the publication locations of that collection. Always check the collection dates and publication title and location prior to starting a search project. 3, Remember you are searching for a string of letters, NOT a Name or a Word For example, if you are looking for a person named "Smith" (good luck with THAT common name, but I digress), try using different letter strings, such as "Smitb", or "sm1th", or "8mith" or combinations of those different letters/numbers. If there is an "h" in your search term, try exchanging a "b", since b's and h's are quite similar and thus the OCR text may include the "b" rather than the desired "h" (e.g., an undesired ink spot in just the right location may make the "h" look like a "b". As a real world example, searching the California Digital Newspaper Collection for the surnames "Braunhart" yields 1,507 results. Replacing the "h" with a "b", hence searching for "Braunbart" yields 96 results - for the SAME person. That is approximately another 6%! For a similar reason as "h" and "b" are confused - the same holds true of "c" and "e". Likewise, lower case m's and n's are often confused. The m's are often converted to several combinations of letters. Also r's and n's can be confused. For a complete list go to The Best Way to Find More Pertinent Articles in Historical Newspaper Research To repeat - you may "wish" that when you enter a perfectly spelled name or word, that you will get positive results when searching against the OCR text - but actually it is the string of letters that appear in the text representation of the page AFTER the OCR process that you are searching against. So, outsmart the system by changing your search string of letters. 4. Remember to Take Hyphenated Words into Consideration Hyphenated words were often used because of fixed width type as well as the experience and capability of the typesetter. Hyphens are less utilized today but were a staple years ago. Take that into consideration if you are searching for a surname or other search criteria with many letters in one word. Try splitting the search into two words where the hyphen may have been normally used. I have seen in older newspapers, as much as 20% of a column have hyphenated words. So try splitting up someone's name. For example, MacDonald could have "Mac-" on one line, followed by "Donald" on the second line. Or MacDon-" on one line and "ald" on the second line. You will get many more positive results if you take hyphenation into consideration when formulating your search string. Here is more about searching for hyphenated words - Use Hyphenated Search 5. Use Crowdsourcing to Improve the OCR Textual representation of the Page More and more digitized collections are using crowdsourcing to improve the OCR text. In effect you are helping the next person by correcting OCR errors and changing the letter string to words or names that are spelled correctly. Additional help for all our searching success would be more crowdsourcing to correct OCR errors and improve the text. An example is reCaptcha processing that has been used by Google Books. Another crowdsourcing example that I personally have used is that of correction on the actual online newspaper site, such as the aforementioned California Digital Newspaper Collection. In this example, registered users can provide edited text that is then incorporated into future searches. Kind of like a newspaper-related "pay it forward." This capability is provided on that site and many others from the fine folks at Elephind.com, who created the software used by the California collection as well as many other online newspaper sites. I have recently noticed more software that is incorporating this feature. So if you help improve the text index and others do as well, everyone benefits! So, in summary - you can win with online newspaper searching - you just need to be smart about the process. Good luck - be persistent and have reasonable expectations. Finding information about historical events or finding an ancestor in newspaper articles is an art, as well as a science. One must be clever and resourceful to get around the limitations in old newspapers, whether they be poor original quality, copies of copies as sources for scanning, as well as the limitations of the OCR process. Another "feature" of older newspapers is the use of the hyphen. Hyphenated words were often used because of fixed width type as well as the experience and capability of the typesetter. Hyphens are less utilized today but were heavily used years ago. Let’s look at a paragraph or two from a newspaper over 100 years old and notice the abundance of hyphenated words at the end of several lines: The example above is for two paragraphs, where there is a total of 22 lines and 6 of them contain hyphens. That is a large percentage of the lines. What that means is that you are not obtaining the maximum number of search results if you do not spend the time to search for parts of hyphenated words.
The suggestion is that if you search for a portion of a surname (if it is multi-syllabic), rather than the entire word, you may get additional results. For example, if your desired name was "Jorgenson" try searching for "Jorgen." The typesetter may have split the word so that at the end of one line are the letters "Jorgen-" with the hyphen, and the next line may start with "son". I have ancestors with the surname "Braunhart". Many times, an article may have a line that ends with "Braun-" and the next line starts with "hart". There can be additional challenges though, as "Williamson" may be split up as "William" and "son". Both sets of letters end up being very common words, so your results may be too numerous to be of much help. Even a short word like "others" can be split up, so you would search for "oth". This may lead to too many results but is worth a try. And always consider that the "break" may not always be in the logical place in the word. Be open minded about where the split may occur. In the case of McDonald, I wouldn't think that the name would be split with "McDon" and "ald", but searching for odd splits may lead to many more results. Don't give up; let hyphens give you additional opportunities and additional searches. You will indeed find more articles this way. Be creative! Don't be bogged down by 21st century logic. Also, remember that the split may be different than you would think it should be. Whether you are searching old newspapers to find an ancestor as part of your genealogy research, or you are searching for an event in these newspapers as part of a classroom assignment, or for history research, these tips all apply.
Many times, we get stuck in searching just names, but there are excellent alternatives that will help you hone in on articles that apply to your research. If you are searching for a person, try these different types of searches combined with a name, or by themselves in certain cases:
If you are searching for an event, some of the above tips may apply, but also:
The point here is to “think outside the box.” Using information of all types that you know or suspect about the person or event will help you uncover more relevant newspaper articles. When entering our search criteria in the search box on an online historical newspaper site, we often focus primarily on the name of the target person. We might also search for an event, or a location or an address. or combine these with a name. I offer you some additional ways to hone in on target person articles, and this is to use keywords in your search criteria. Here are some examples that are should be considered, that if used will get you more and better results: If you are searching for obituaries, add the keyword "beloved" or "beloved son" or "beloved wife" etc. The word beloved is used frequently in obituaries. So, entering these keywords plus the surname may focus in on obituaries.
These examples are especially useful for genealogical research, but should be useful for other pursuits. There are likely many more examples. Try adding keywords - you will be glad you did. In old newspapers, abbreviations were used to save space. Abbreviations were used to shorten many words - the most notable being given names as well as addresses. Remember that your search criteria is just a series of letters that are matched against a search index that was created from the scan and OCR process applied to the subject newspaper page. Your search index database is not smart enough, for example, to interpret "ave." for the word "avenue." So, if you are entering "335 14th Avenue" - if the search index includes "Ave" the search will not be successful. Therefore, you must search both "335 14th Avenue" and "335 14th Ave" to get all possible results. Here are some examples of abbreviations that you should employ in your search criteria:
There may be many others, so you need to search for abbreviated words as well as entire words to maximize your search results. My guess is that you can increase your pertinent results by as much as 10%. Other than making sure that you are searching in the right place geographically and using a date range that fits your ancestors lives, what follows is the ABSOLUTE BEST WAY to find more pertinent articles when searching online historical newspapers.
If you do newspaper research online as part of your history research, then you have certainly been puzzled by some of the search results (or lack thereof) that you have received. Creation of newspaper images and the application of the OCR process does not always result in what you might expect in the index which is used to match against your search criteria. There is a simple explanation for this, and it all has to do with quality:
Thus what you are searching probably won't be a perfect digital database that represents what was originally written by the author and newspaper publisher. What can we do about it? There are lots of things to try and this article deals with changing the letters in your search criteria. For example - if a surname you are searching for is "Wilson" and the letter "n" is often converted to the letter "m" in the index from the OCR process, why not search for "Wilsom"? I guarantee that changing your search letters will lead to an improvement of at least 5 to 10% in search results. I heard from one reader that changing letters and letter pairs got them a 20% improvement! So what letter pairs are often confused and exchanged in the index?
My suggestion? Change your search criteria and exchange the letter string you are entering in the search box to include these alternative letter and letter pairs and see what happens. In other words - deliberately misspell the name or word. You will be pleasantly surprised! No matter what your search criteria represents - a name, an event, a city name, etc. to the OCR process it just comes to a series of dots that make up a random string of letters. For other ways to improve your search results and to learn much more about becoming an expert newspaper researcher, check out the Tips page. |
SEARCH
Archives
January 2022
Categories
All
|