I'm trying to parse all XML files in a given directory using python. I am able to parse one file at a time but that would be 'impossible' for me to do due to the roughly 19k different xml files, i.e. it works when I pre-define the tree and the root, however not when I try to run for all the code.
This is what I implemented so far:
As a result, I used try...except as advised in the prior article by @wundermahn and @Kevin. The current output is this. For example, 482 out of 19320 goods have a problem, yet the apparently good items aren't "functioning." Currently, I'm having trouble printing out a specific piece from the list post list[. I'm experiencing the following issue:
I also checked post list and printed it out, and for some reason there is no data there and it is empty.
Sample XML:
Further debugging revealed that the problem appears to occur whentree= ET.parse(filename) is called, but I am still unable to identify the problem.
This is what I implemented so far:
Code:
import xml.etree.ElementTree as ET
import os
directory = "C:/Users/danie/Desktop/NLP/blogs/"
def clean_dir(directory):
path = os.listdir(directory)
print(path)
for filename in path:
try:
tree = ET.parse(filename)
root = tree.getroot()
doc_parser(root)
except:
print("ERROR ON FILE: {}".format(filename))
post_list = []
def doc_parser(root):
for child in root.findall('post'):
post_list.append(child.text)
clean_dir(directory)
print(post_list[0])As a result, I used try...except as advised in the prior article by @wundermahn and @Kevin. The current output is this. For example, 482 out of 19320 goods have a problem, yet the apparently good items aren't "functioning." Currently, I'm having trouble printing out a specific piece from the list post list[. I'm experiencing the following issue:
Code:
IndexError: list index out of rangeSample XML:
Code:
<Blog>
<date>05,July,2003</date>
<post>
B-Logs: The Business Blogs Paradox urlLink HindustanTimes.com discusses the effects of technology and blogs in particular. According to the article, Blogs are 'a direct, one-to-many vehicle for communicating ideas'. What makes them disruptive in business application is that they allow businesses - which are after all human -- to communicate with a human 'real voice'. On the other hand, urlLink Webpronews.com discusses the idea of corporate newsletter publishing via blogs. I found the idea very pragmatic and futuristic. Way to go !!
</post>
<date>04,July,2003</date>
<post>
Bohemian Rhapsody : Is it?? I have just come back to my room from the urlLink IIT coffee shack. The place, where we have shared laughters and joys, sat for long hours over countless cups of coffee and maggi and chat sessions. Where we have shared our thoughts and fought over stupid topics discussing them passionately. I have just come back from that very place, but with thoughts and feelings so different from what it used to be. Today the laughter has given way to a look of tension and frustration on everyone's face. "Its so frustrating. There is no job. I am applying everywhere. No vacancies." And then on the next table you hear, "I am planning to go abroad on a scholarship. The job scene is really bad and I don't think I will get a nice job. Going for higher studies is the only option left in front of me." And then a person comes who does have a job. Meeting him, I congratulate him. But he is too irritated with the job he got. The job is in a core engineering one, in a chemical MNC, but the pay is a punitive six thousand bucks; the only perk being free accomodation in some small village-town. There are ruminations galore, about the uncertain futures. About the decisions made. About the hard work, before and during IIT years. And to end thus, with a degree in hand but no job to do justice to years of hardships and hardwork, and that too when you are the product of one of the best institutes in the country. It gets too disappointing. I only wish those people, my dear friends, good luck !! May serendipity happen !!
</post>
<date>03,July,2003</date>
<post>
Entrepreneur Guidelines urlLink EntreWorld is a website for entrepreneurs. It is a resourceful site for budding start-ups, discussing various dos and donts and giving practical advise for all stages of a start-up life cycle. urlLink Auren Hoffman 's recent article on Entrepreneurial Boards discusses the leadership engine for the evolutionary stages of a start-up. "Boards of advisors are best for helping entrepreneurs build companies in the formative stage, whereas boards of directors lend a hand during times of crisis or change, writes a serial entrepreneur." Read the full article urlLink here . A strong board of directors can make the difference between success and failure for a start-up company. And, an effective advisory board can help one recruit those directors. Specific examples, indicating the tasks a board can accomplish and the skills it can bring to them, make this article particularly enlightening. The urlLink article can be assumed to be a Business 101 course :)). Further, here is a urlLink business plan guide.
</post>
<date>02,July,2003</date>
<post>
Business Profiles of Some Companies urlLink Business Profiles Home Page makes an informative reading. Biz/ed have created profiles for a range of business organisations. These are presented as sets of FAQs, grouped into themes.The companies profiled vary from McDonlad's to easyJet. *How much does it cost to set up your own pizza delivery franchise? *What's the best way of promoting online gambling? *How do the major supermarket retailers compete? *Is it possible to run a professional football club as a private investment? Answers to these questions and many more can be found on the urlLink Site .
</post>
<date>02,July,2003</date>
<post>
Multiple Interviews for One Job: The present-day scenario. urlLink csmonitor.com published an article on the 12-step job interview. There was a time when the recruiter took 20 minutes to size you up and skim your résumé before leaping to his feet and barking, 'Kid, I like the cut of your jib. Welcome aboard!'. It makes an interesting reading. But today, with employers comfortably ensconced in the labor-market driver's seat, hiring decisions based on instinct are practically unheard of. Indeed, it's not at all unusual these days for a candidate to be grilled by six, eight, or even a dozen interviewers on various rungs of the corporate ladder as part of the overall screening process, say human-resources executives, headhunters, and other experts in modern hiring practices "Many companies have made bad hires; now it's their market, and they're determined to find the people they want," explains Marie Raperto of the Cantor Concern, a New York City recruiting firm. "Even someone seeking a mid-level job has to be prepared to go through six or seven interviews," she adds. "It's endless." "You can't even get nine people to agree on where to go for lunch," he adds. "How can you expect them to agree on a person?" But in an age where companies routinely boast about their teamwork ethos, nonhierarchical cultures, and commitment to "cross-functional" collaboration, it's easy to see why consensus now plays a major role in hiring. Many companies known for attracting top-flight talent say they believe that gathering a wide variety of perspectives is essential to ensuring that the right person gets the job. Problems with group hiring arise when junior staffers or peers are given veto power in the final decision, says Bob Woodrum, a partner at executive-recruiting giant Korn/Ferry International. Recounting a recent incident in which a candidate was dismissed by a Fortune 100 client despite having favorably impressed 11 of 12 interviewers, he notes that "everyone has a different agenda," and that such agendas - whether personal or political - can conflict with the organization's best interests. "This was a case where 11 people had said, 'This guy's a hire.' But one person said the candidate wasn't enthusiastic enough, and that was it." Read the whole article urlLink Here .
</post>
<date>02,July,2003</date>
<post>
Google Prank urlLink Google has played yet another neat prank. Not sure how long this will last though. So, do rush to Google home page, type "Weapons of Mass Destruction" and hit the "I'm Feeling Lucky" button. Read the error message text CAREFULLY !!
</post>
<date>02,July,2003</date>
<post>
Origami Boulders I came across this hilarious site selling Origami Boulders. Origami is a Japanese art of folding paper. What they are selling is wadded up paper or something like that. The amusing part is the sarcastic way. Interesting are the dealings with the post office and banks. The site promises free shipment (though God only knows who would like to order wadded up paper). Its a laughing riot all the way. Check it out. The site title bears the name urlLink Origami Boulder Company -- Original Origami Gifts! . Check out this excerpt from the site: "Site is real. You order and you really get origami boulder artwork with special card to display at your home or workplace. Make good unforgettable gift for friends! You buy wadded paper boulder and keep it. Or send many to your friends as very nice gift that no one ever forget! I include special card with every order that explain work of art. You buy 20, I send you free extra one with special message from me! Hurry up and order now! I send you wadded paper with Priority Mail. It is fast with nice free box from Post Office. Post office worker tell me, 'Don't take so many free boxes! You must order them online from USPS! Other customers mad when you take them all!' I laugh and yell, 'It says free, bureaucrat!! What you expect, dumb dumb?' Update!!!! Wall Street Journal article on 5/29/02 say that Priority Mail is ripoff and doesn't arrive faster than First Class mail. This is outrage from post office lazy people. I complain today at post office and they laugh and pretend article isn't true. Who you believe, slow postman or Wall Street Journal? Now maybe I buy special boxes and send First Class instead of wasting money on Priority Mail. I make most efficient decision for customer benefit. My friend is graphic designer for big company. She design page for me because FrontPage too hard for wadded paper artist! I change design and she send email that says, "you've ruined my beautiful site!!! :O( what in the hell is up with origami boulder?!?!?!?!" I tell her, "You designer, not site owner! I change whatever I like. You get paid, didn't you? Then go away now!!!" Her design have too many pages and Jakob Nielsen say Internet people too lazy to click so I make everything one page.
</post>
<date>02,July,2003</date>
<post>
Seth Godin Came across the blog site urlLink Seth's Blog . Seth writes in an iconoclastic manner, complete with tangential thoughts and lateral humor; thus living up to his strature of the ultimate entrepreneur of the year and a marketing guru. I think this blog is going to be added in my everyday-visits site list. Check it out. Its amazing. Herez an article from his site, that he posted: Naming a business Greg Harrington writes, "I’ve been thinking quite a bit about a topic lately—how to best name a business—and in looking for some ideas, I’ve reviewed several of your books, but don’t find anything in the way of a thorough treatment of this topic." Here's what I think: First, the main point: a brand name is a peg that people use to hang all the attributes of your business. The LESS it has to do with your category, the better. If you call yourself International Postal Consultants, there's a lot less room to hang other attributes. Some names I like? Starbucks. Nike. Apple. Second, please pick a real english word, or a string of them. Axelon and Altus are bad. Jet Blue, Ambient and Amazon are good. Third, be sure it's easy to spell AND pronounce. Prius is a bad name. I can't tell anyone to buy a Prius because I'm embarrassed I'll say it wrong. Fourth, don't obsess about getting a short web name. If you want to name your venture capital firm Nickel (a great name, imho) then you could have www.NickelVenture.com and that would be fine. The only way this turns into a problem is if the current owner of the URL is a competitor (which won't happen if you pick a non-obvious name, as I write in #1 above). If you follow these pieces of advice, you'll discover that there are literally millions of names available to you (lemonpie, for example, is perfect for a scuba tour company. So are orangepie, melonpie and kiwipie). You will have far fewer trademark hassles. You will have no trouble coming up with a cool name that means nothing and makes it easy for you to hang a good brand upon. And you'll have fun. BUT, don't forget to come up with a great tagline. "lemonpie, the easy way to learn scuba," for example. PS a couple more tricks: 1. Use a stock photo CD and find cool pictures that match your name BEFORE you pick the name. If you can find a bunch of $30 images that work with a name, grab the pictures, then the name. 2. Don't listen to anyone else. All your friends will hate it. GOOD. They would have hated Starbucks too (you want to name your store after something from Moby Dick!??) If your friends like it, run.
</post>
<date>02,July,2003</date>
<post>
Schools of the Future urlLink Ode Magazine's latest issue sports an article written by the famous Ben Okri on the "Schools of future". He writes that in the future, centres of learning will teach at least one thing we do not teach today: the art of self-discovery. There is nothing more fundamental in education. We turn out students from our universities who know how to give answers, but not how to ask questions. A really well written article. I hope we would analyse the shortcomings mentioned there, and strive to improve the educational models we currently have, imbibing new ideas and new paradigms into our present systems.
</post>
<date>02,July,2003</date>
<post>
IT Firms Slash Entry Level Salaries urlLink Economic Times reports that though most companies are moving towards performance-linked compensation packages or a variable pay model, industry estimates suggest that entry level salaries have been cut by 20-25% across major companies. Even as this happens, companies are rewarding top performers by increasing their pay. “While companies in the US selectively reduce salaries to fund the bonus pool, companies in India are also following suit,” said a CFO at a leading Indian software company. Though it is not known whether a 20-25% cut in entry level salaries would be enough to fund bonus reserves and other allowances, sources in top rung companies state that since the measure will be across the board, it would lead to substantial benefits. The market is looking forward to the first quarter results, beginning with MphasiS. Infosys is slated to announce its results the next day. “All eyes will be on Infosys, which has taken strong measures to further cut costs. These measures, which include cuts in entry level compensation, onsite rationalization of salary-related allowances and a focus to shift work offshore, would reflect in the coming quarters. But the cost-push pressures from offshore salary hikes could be more immediate, when seen together with a weakening dollar and rate declines,” an analyst at Credit Lyonnais Securities said in his report. Read the full report urlLink Here
</post>
<date>02,July,2003</date>
<post>
</Blog>Further debugging revealed that the problem appears to occur whentree= ET.parse(filename) is called, but I am still unable to identify the problem.