Ben Nagy, off the firewalls list had a very concise and strong response to such info, further supporting some of H Carvey's assurtions here on this topic, I'm reposting it here: From: Ben Nagy <ben.nagyat_private> Subject: RE: Firewall testing recommendations Cc: firewallsat_private Date: Mon, 10 Sep 2001 15:55:43 +1000 To: 'Paul D. Robertson' <probertsat_private>, Daniel Crichton <danielcat_private> But any "analysis" process should include external verification - ie that the box is doing what you told it to do, right? This is quite distinct from the traditional pen-test in that it isn't blind. I think that to create the most secure system possible, blind pen-testing is a waste of time - this seems to align with Paul's ideas. I still see value in "normal" (blind) pen-testing, mainly for risk assessment purposes, in other words they're good for working out if the holes you left open as as small as you thought they were. I'm a strong believer that systems don't need to be perfect - one just needs to know quite accurately how imperfect they are. I'm going to call Paul's idea Full Disclosure Penentration Testing. Procure one or more experts, show them all your configuration environment, let them ask as many questions as they want, and then ask them to verify that everything is operating how it should be. This should be useful in detecting software faults which cause vulnerabilities even with sound configurations. The way I see it, the ultimate goal of the FD Pentest is to break systems. Contrast this with the goal of the blind pentest, which is to find out if systems will break under a fairly "normal" attack scenario. One is good for building very secure systems. The other is good for convincing yourself that you've spent enough money to have a defensible risk position. I like to talk about Visa here - they know that they will suffer lots of fraud, and they could make it harder, but they have determined that they still make lots of money and that customers don't want more stringent card security methods. Maybe "Verification Testing" would be a better name than FD Pentesting. Hm. Dunno. Anyway. -- Ben Nagy Network Security Specialist Marconi Services Australia Pty Ltd Mb: +61 414 411 520 PGP Key ID: 0x1A86E304 > -----Original Message----- > From: Paul D. Robertson [mailto:probertsat_private] > Sent: Friday, September 07, 2001 10:57 PM > To: Daniel Crichton > Cc: firewallsat_private > Subject: Re: Firewall testing recommendations > > > On Fri, 7 Sep 2001, Daniel Crichton wrote: > > > I'm looking at external penetration testing for my firewall and was > > wondering if > > anyone has any recommendations for good tools or services > to do this. I'm > > As always, it's probably a good thing to question what you > want to do and why before you decide to do it. > > Quite a lot of companies make quite a bit of money doing > penetration tests. I've always been of the opinion that pen > testing is significantly less useful than local configuration > checking and validation by an expert. [...] > good analysis > is almost always more valuable than great penetration testing > and great analysis wins every time. > > Paul _______________________________________________ Firewalls mailing list Firewallsat_private http://lists.gnac.net/mailman/listinfo/firewalls On 11 Sep 2001, H Carvey wrote: > > > A zero knowledge pen test > > should be the starting point of an audit, > > I couldn't disagree more. A pen test is not an > audit. An audit, by it's very nature, assumes > that the information infrastructure is being > compared to some standard. Ideally, this standard > will be supplied by the available corporate > information security policies. Absent these (as > is the case many times), some other more arbitrary > standard must be used. This alternate standard > should be based on the consultant's understanding > of the business needs of the client. > > The entire information infrastructure should be > examined in this audit (what I referred to in an > earlier post as a vulnerability assessment). This > includes informal and undocumented procedures and > processes, as well as actual host settings and > configurations. > > A pen test, by it's very nature, only focuses on a > relatively small portion of the infrastructure, > that being the public interface of the > organization. So much more information can be > learned by examining as much of the infrastructure > as possible (including interviews of key > personnel, etc). This data is then analyzed in > the context of the client's business, and then the > final deliverable presents that analysis in a > manner that is useful and pertinent to the client. > > The only real usefulness of a pen test is to test > the reactions of the incident response team. If a > router ACL or host configuration setting is > applied and verified, you don't then need someone > to try and break in from the outside just to > verify it again. > > Further benefits of a vulnerability assessment > include the knowledge transfer that goes on > between the consultant and the client. The > consultant does no one any good if he disappears > into a room and produces a magical report in a > vacuum. Consultants must work closely with > sysadmins at the client site...after all, after > the consultants are gone, the sysadmins are left > with whatever's there, whether they understand it > or not. Besides, working closely with the admins > builds trust and adds credibility...which leads to > a relationship and follow-on work. > > > Regarding studies like CSI/FBI survey, > > For the sake of the readers and the moderators > alike, I will refrain from my usuaul diatribe > regarding quoting such things as this survey...and > particularly this survey. > > > more or less, the 1st test will cover > > about 20-30% of the potencial attackers while > the 2nd will cover the others > > 70-80%. > > I would argue that a well-planned vulnerability > assessment, if delivered properly, and if the > necessary changes are made, will result in > protecting the client from 90% or more of both > internal and external threats. > > I say this b/c the reports I have delivered stress > the need for policies, and for recognizing that > security is a process, that must be continually > maintained. > > > The 1st test should be much longer in time and > resources, and usually the > > clients here don't understant quiet well where > their money goes. > > So why should they pay it? > > > > So most of > > the times clients prefer to contract the 2nd > test only, because it takes > > less time and money. Also, that's why after that > their systems are still > > vulnerable. > > That doesn't make any sense. Even if the second > test is contracted only, the final deliverable > should be clear enough such that if the documented > issues are addressed, the client won't be _as > vulnerable_ as they were. > > > It is important for the client that a little > education is provided, > > Again, I disagree. It is important for a client > that _a lot_ of education is provided. This means > not only up front during the sales process, but > also during the assessment, and afterward, when > the deliverable is presented, as well. > > > And also, > > why pen tests should be regular, each month or 2 > or 6 or whatever ... > > I agree with regular testing...but is it really > necessary? See below... > > > An > > audit will only cover a specific period of time, > so it is not anyway and not > > anyhow a garantee that in the (short) future > problems will not happen. > > Of course not. That goes without saying for the > security professionals, and should be clearly > communicated to the client. However, if the > consulting firm does it's job and adequately > communicates the concept that security is an > ongoing process ("Mr. Client, you need to install > these patches, and in the future you may also need > to install others...as they come out."), then it's > in the clients hands. At that point, if they > choose not to follow the advice...so be it. > > > At the technical side and at the commercial side > for both parts (consultant > > and client), the more audits in a period of time > are made, the better the > > investiment and the results. > > That's not necessary. The disruption to a > client's infrastructure is unacceptable. The > appropriate way to conduct these things is to > conduct a comprehensive vulnerability > assessment...planned so as not to engage any one > portion of the infrastructure for too long...and > provide the recommendations in the deliverable. > The client can then map out a timeframe for > completing what he feels are the necessary steps, > as communicated by the consulting firm. This > could be 6 months, or a year. Re-examining the > infrastructure after a specified period is advisable. > > Remember, it's the client that drives the boat, > not the consulting firm. The client pays, > therefore the golden rule applies. > > > Some security consulting companies will not give > away clients references, > > because keeping confidential other companies > with past or present security > > problem is part of the contract with a client. > > Security companies that provide references (and > many do) do so only after their clients approve. > Simply b/c a security consulting company does an > audit doesn't mean that the client had a "security > problem" at all. In fact, many companies use > comprehensive audits/assessments performed by big > named firms to show due diligence. > > Basically, the way it works is this...security > company "A" asks client "B" if they can use the > client as a reference. Client "B" says yes, with > stipulations that the exact nature of the work not > be described ("we did a security assessment for > client B"). Then potential client "C" calls > client "B", and client "B" says, "yes, these guys > do good work." Simple, yes, but that's basically it. > > If someone had a "security problem" (i.e., an > "incident")...no, of course the wouldn't want it > publicly known. But the security firm that did > the work wouldn't ask, either. > > > Other problem can be also giving away > > models and methods, because there are many > smartasses that are just looking > > after knowledge to do the job themselfs. > > By "smartasses", I guess you mean "potential > clients". Yes, these folks are out there...I've > seen them, as I'm sure others have. Yet there is > nothing to prevent either party from requiring > NDAs before going forward. > > > ---------------------------------------------------------------------------- > This list is provided by the SecurityFocus Security Intelligence Alert (SIA) > Service. For more information on SecurityFocus' SIA service which > automatically alerts you to the latest security vulnerabilities please see: > https://alerts.securityfocus.com/ > -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ admin & senior consultant: darkstar.sysinfo.com http://darkstar.sysinfo.com "Cutting the space budget really restores my faith in humanity. It eliminates dreams, goals, and ideals and lets us get straight to the business of hate, debauchery, and self-annihilation." -- Johnny Hart testing, only testing, and damn good at it too! ---------------------------------------------------------------------------- This list is provided by the SecurityFocus Security Intelligence Alert (SIA) Service. For more information on SecurityFocus' SIA service which automatically alerts you to the latest security vulnerabilities please see: https://alerts.securityfocus.com/
This archive was generated by hypermail 2b30 : Wed Sep 12 2001 - 14:37:19 PDT