WAL_MART

Wal-Mart was the victim of a serious security breach in 2005 and 2006 in which hackers targeted the development team in charge of the chains point-of-sale system and siphoned source code and other sensitive data to a computer in Eastern Europe, Wired.com has learned. Internal documents reveal for the first time that the nations largest retailer was among the earliest targets of a wave of cyberattacks that went after the bank-card processing systems of brick-and-mortar stores around the United States beginning in 2005. The details of the breach, and the companys challenges in reconstructing what happened, shed new light on the vulnerable state of retail security at the time, despite card-processing security standards that had been in place since 2001. In response to inquiries from Wired.com, the company acknowledged the hack attack, which it calls an internal issue. Because no sensitive customer data was stolen, Wal-Mart had no obligation to disclose the breach publicly. Wal-Mart had a number of security vulnerabilities at the time of the attack, according to internal security assessments seen by Wired.com, and acknowledged as genuine by Wal-Mart. For example, at least four years worth of customer purchasing data, including names, card numbers and expiration dates, were housed on company networks in unencrypted form. Wal-Mart says it was in the process of dramatically improving the security of its transaction data, and in 2006 began encrypting the credit card numbers and other customer information, and making other important security changes. Wal-Mart really made every effort to segregate the data, to make separate networks, to encrypt it fully from start to finish through the transmission, says Wal-Marts Chief Privacy Officer Zoe Strickland. And not just in one area but across the different uses of credit card systems. Wal-Mart uncovered the breach in November 2006, after a fortuitous server crash led administrators to a password-cracking tool that had been surreptitiously installed on one of its servers. Wal-Marts initial probe traced the intrusion to a compromised VPN account, and from there to a computer in Minsk, Belarus. The discovery set off an investigation that swept in outside security consultants and corporate attorneys to determine what the hackers had touched, and whether the company was required to report the intrusion, and to whom, the documents show. Wal-Mart says it notified federal law enforcement agents, who were working on other ongoing investigations involving similar breaches. At the time, attacks featuring a similar MO were occurring at TJX, Dave & Busters restaurants and other companies, which ultimately resulted in more than 100 million cards being compromised. Albert Gonzalez, a 28-year-old Miami man, pleaded guilty this month to carrying out many of those breaches with other hackers, and is facing unresolved charges for the remainders. The Wal-Mart intrusion began unraveling on Nov. 5, 2006, when the companys IT security group was brought in to investigate the server crash. Wal-Mart has thousands of servers nationwide, and any one of them crashing would ordinarily be a routine event. But this one raised a red flag. Someone had installed L0phtcrack, a password-cracking tool, onto the system, which crashed the server when the intruder tried to launch the program. Investigators found that the tool had been installed remotely by someone using a generic network administrator account. The intruder had reached the machine through a VPN account assigned to a former Wal-Mart worker in Canada, which administrators had failed to close after the worker left the company.

The day the server crashed, the intruder had been connected to Wal-Marts network for about seven hours, originating from an IP address in Minsk, the documents show. The security team disabled the compromised VPN account, but the intruder, who should have realized the jig was up, came back in through another account belonging to a different Canadian employee. When that VPN account was closed, the intruder grabbed yet a third account while Wal-Mart workers were still scrambling to get a fix on the scope of the breach. When Wal-Mart reviewed its VPN logs, it found that the activity had begun at least as early as June 2005, according to memos written by Wal-Mart employees during the initial stage of the investigation. The companys server logs recorded only unsuccessful log-in attempts, not successful ones, frustrating a detailed analysis. Wal-Mart declined to respond to questions about the initial date of the attack, the server logging or the conclusions it reached in its final report, which Wired.com has not seen. Nonetheless, Wal-Marts security team was able to identify over 800 machines that the attacker either tried to brute force or actually made a successful connection, according to a Nov. 10, 2006 e-mail summarizing the early investigation. Many computers the hackers targeted belonged to company programmers, the documents show. Wal-Mart at the time produced some of its own software, because the company couldnt find off-the-shelf applications that scaled to its size, according to a source who was involved with the investigation. One team of programmers was tasked with coding the companys point-of-sale system for processing credit and debit card transactions. This was the team the intruders targeted. They werent port scanning, they werent ping tracing, they werent groping blindly in the dark trying to find a nugget, says the investigator. They knew what they were going for and they were all over it point-of-sale. The intruders interest in Wal-Marts point-of-sale system is consistent with large data breaches that occurred at other companies around the same time. In the spring of 2005, associates of TJX hacker Albert Gonzales hacked into the point-of-sale system of a Marshalls clothing store in Minnesota. The hackers pointed an antenna at the store to grab data as it streamed over the stores vulnerable Wi-Fi network, then used the data to gain access to the central transaction database of TJX, Marshalls parent company. Similarly, in mid-2007, Gonzalezs gang gained access to point-of-sale servers at Dave & Busters restaurants and installed packet sniffers to siphon card data as it was transmitted to corporate computers and others for authorization. According to court documents, the hackers MO included doing reconnaissance of retailers to determine the point-of-sale systems they used and map their network setups. (Theres no evidence Wired.com has seen linking Gonzalez to the Wal-Mart breach.) In the case of Wal-Mart, one of the documents that flew off to Minsk from a programmers machine was titled POS Store Systems Technical Specifications TLOG Encryption and Financial Flows Draft 03/04/2006 essentially a flow chart that would have mapped out Wal-Marts transaction process, the source says, from the moment a customer swipes his credit or debit card in a stores card reader, to the point the digital data crosses the network to be authenticated by a card issuer. The hackers also stole or accessed files containing point-of-sale source code and executables, as well as additional proprietary documentation detailing the companys transaction processing network. A partial list seen by Wired.com includes documentation on a company database, a file connected to a point-of-sale simulator, debugging files, a telnet capture, a bash history file and a sign-on log.

The documents show no evidence that files containing customer information were breached in the attack. At the time Wal-Mart discovered the breach, it had been encrypting its transaction data for at least three months. It began to do so after a security audit performed for the company in December 2005 found that customer data was poorly protected. Wal-Mart commissioned the probe from security auditors at CyberTrust as part of its efforts to become compliant with Payment Card Industry (PCI) security standards that were established in 2001. Enforced by credit card issuer Visa, top-tier companies such as Wal-Mart were theoretically required to be in compliance with the standards by mid-2004. Wal-Mart says it received a number of deadline extensions. CyberTrust examined networks at five Wal-Mart locations: three Wal-Mart stores in Missouri and Oklahoma, and two other Wal-Mart-owned businesses a Sams Club store in Missouri and a Neighborhood Market in Arkansas, according to a report the auditors wrote. The assessment lasted six days, during which CyberTrust found numerous problems. Each of the five stores, for example, housed complete backup copies of transaction logs on network-connected UNIX servers, which included at least four years worth of unencrypted credit card numbers, cardholder names and expiration dates from purchases at the stores. The auditors also discovered that servers, transaction processing systems, and other network-connected devices handling sensitive information used the same usernames and passwords across every Wal-Mart store nationwide. In some cases, the passwords could be easily guessed. A hacker or malicious insider who compromised a point-of-sale controller or in-store card processor at one store, could access the same device at every Wal-Mart store nationwide, CyberTrust wrote. Finally, CyberTrust found sensitive customer information stored unencrypted on pharmacy computers at four of the stores, including customer names, home addresses, Social Security numbers, genders, credit card numbers and expiration dates. A long-term, undetected compromise of Wal-Mart RXP system could allow a virtually endless supply of customers names, addresses, and Social Security numbers the basic ingredients for identity theft, CyberTrust wrote in its report. Wal-Mart runs the risk of losing not only the sensitive information, but also their customers hard earned trust, the auditors added. The report was dated Jan. 9, 2006, 10 months before Wal-Mart discovered the breach. Strickland says the company took the report to heart and put a massive amount of energy and expertise into addressing the risks to customer data, and became certified as PCI-compliant in August 2006 by VeriSign. After it discovered the breach in November 2006, the company turned over memory dumps and at least 31 forensic images of machines and servers to Stroz Friedberg, a forensic investigations firm, for further analyses. E-mails exchanged by team members eight days after the intrusion was detected show the company furiously searching firewall and intrusion detection logs for suspicious activity. The e-mails also discuss shutting down the entire Nortel VPN network the intruder used, ordering RSA security tokens to authenticate users to the network, and increasing logging retention on servers. On Nov. 16, one team member sent an unencrypted e-mail update to other employees and was harshly rebuked by a senior security manager who warned them to communicate only through e/pop, a secure instant messaging system. Guys. time out here, he wrote. What was the first thing I discussed in our meeting about communications protocol concerning this project? Get Epop up, installed, and running today!

The companys internal investigators found evidence potentially linking the attack to a suspected breach at a Wal-Mart division a year earlier. The forensic trail showed that the machine in Belarus that breached Wal-Marts VPN had tried to log on to a machine belonging to Sams Club, Wal-Marts membership store chain, in 2005. This finding was potentially significant, because Sams Club had been suspected in 2005 of spilling credit card data in a breach. Late that year, MasterCard and Visa informed Wal-Mart about a cluster of fraudulent charges on credit cards that had been used at Sams Club gas stations. A press release issued by Sams Club at the time warned that intruders might have gained access to 600 cards used at the pumps between Sept. 21 and Oct. 2, 2005. But the company assured consumers that the electronic systems and databases used inside its stores and for samsclub.com are not involved and now says that after investigating the issue, it never found any evidence of a breach at its gas pumps or in its stores. The company also says the Sams Club investigation and the 2006 breach are not connected. The Sams Club matter has been closed for some time and is not related to the other matter youre asking about, said Wal-Mart spokeswoman Michelle Bradford. Internal documents show that Sams Club suffered the same types of vulnerabilities as the rest of WalMarts empire, and that logging was inadequate to completely rule out a breach. An audit by VeriSign at the time found that Sams Clubs firewall and intrusion-detection logs were configured to record only spotty and inconsistent data, and that operating systems lacked the latest security patches. The level of vulnerability identified . . . would leave these systems open to compromise from a number of different attacks, wrote VeriSign in a report. But due to the level of logging enabled on these systems, which did not capture much information, it was not possible to determine if any of these vulnerabilities were attacked. In the report, which Wal-Mart submitted to the Federal Trade Commission in February 2006, nine months before it discovered the breach in its main network, VeriSign concluded that although it had found no point of compromise in the Sams Club system, the companys logs didnt contain sufficient information to identify or rule out a specific point of compromise. Wal-Mart says that the security issues raised in all of these reports have been addressed and that since the company became PCI compliant in August 2006, it has been commissioning PCI audits every six months twice the frequency required by PCI standards. Every item [in the reports] that had a PCI vulnerability was remediated, says Strickland. PCI certification doesnt guarantee the security of bank card data numerous companies that experienced serious bank card breaches in recent years were certified PCI compliant at the time they were breached. There is no evidence, however, that Wal-Mart suffered a sizable breach of credit or debit card data from either Sams Club in 2005 or from its main network in 2006.