To the left, looking north on Main Street from the corner of Curtis, is a long double row of Siberian Elms linking Main and Abel Streets. These were planted in the 1870s by John O'Toole along the driveway to his 20 room Victorian mansion. O'Toole grew produce, hay, and raised thoroughbred race horses here and on acreage near Coyote Creek.

The mansion and the 100+ acres of farmland surrounding it were bought by James Boyd in 1883. He did not keep it for long. In 1884, he sold the property for the then grand sum of $24,000 to Santa Clara County for use as an almshouse. Newspapers of the day called it a "palace for the poor," and other such terms to voice their indignation over what they saw as a waste of public money.

In spite of initial negative press reaction, it was here the county's poor and homeless could find shelter and work on the farm. A portion of what was grown was given to the poor to sell. Many residents of the almshouse set up stands along Oakland-San Jose Highway (now Main Street) to sell produce and wild mushrooms to travelers. In this way those temporarily down and out could earn their keep and put aside a little to get them back on their feet. Poverty was not viewed as a permanent condition in those days. It was believed and practiced that anyone who was willing to work hard could get ahead.

In the 1940s the county began to house low risk prisoners at the site. Over time, more and more of the ranch buildings were torn down to make way for cell block construction. In 1962, eight elderly men, the last Almshouse residents, were evicted and the historic Victorian mansion was demolished to make room for increasing numbers of prisoners to be housed at the County Jail. The name of the jail became Elmwood because of these stately trees. Today, there remains only one of the buildings from the agricultural days of Elmwood's history left standing. It is called the "library" and public access is prohibited.

In the late 1970s the Fire Chief, alarmed over a large branch breaking off in a wind storm, ordered all of the trees topped and stripped of their leafbearing branches. The upper third of the treeas was removed. Although nearly all of the ancient arbor survived the assault, the cuts were left parallel to the ground. Rainwater puddling on this surface has cause rot and allowed insects to invade the trees.

With new housing developments being constructed to the north and south of the O'Toole Elms, the City Council acted in 2005 to destroy these stately reminders of our past to make construction more convenient for the builder.

The last bridge over the old Penitencia Creek bed was located at the east end of the trees just a few feet from Main Street. The north end of the culvert bore the date 1905. The fire department removed the culvert to make room for storage some time around 2006.   It is difficult for anyone now to experience sitting on the concrete abutments like the poor of a past century did while selling their vegetables.