Loading...
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 | /* * Copyright (c) 2012 Apple Computer, Inc. All rights reserved. * * @APPLE_OSREFERENCE_LICENSE_HEADER_START@ * * This file contains Original Code and/or Modifications of Original Code * as defined in and that are subject to the Apple Public Source License * Version 2.0 (the 'License'). You may not use this file except in * compliance with the License. The rights granted to you under the License * may not be used to create, or enable the creation or redistribution of, * unlawful or unlicensed copies of an Apple operating system, or to * circumvent, violate, or enable the circumvention or violation of, any * terms of an Apple operating system software license agreement. * * Please obtain a copy of the License at * http://www.opensource.apple.com/apsl/ and read it before using this file. * * The Original Code and all software distributed under the License are * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES, * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT. * Please see the License for the specific language governing rights and * limitations under the License. * * @APPLE_OSREFERENCE_LICENSE_HEADER_END@ * * This file implements the following function for the arm64 architecture: * * int strncmp(const char *s1, const char *s2, size_t n); * * Returns 0 if the two strings are equal up to the first n bytes or to the * end of the string, whichever comes first. Otherwise, returns the difference * of the first mismatched characters interpreted as uint8_t. */ #include <arm64/asm.h> .globl _strncmp /***************************************************************************** * Macros * *****************************************************************************/ .macro EstablishFrame ARM64_STACK_PROLOG stp fp, lr, [sp, #-16]! mov fp, sp .endm .macro ClearFrameAndReturn ldp fp, lr, [sp], #16 ARM64_STACK_EPILOG _strncmp .endm #include "../mach/arm/vm_param.h" #define kVectorSize 16 /***************************************************************************** * Constants * *****************************************************************************/ .text .align 5 L_mask: .quad 0x0706050403020100, 0x0f0e0d0c0b0a0908 /***************************************************************************** * Entrypoints * *****************************************************************************/ _strncmp: EstablishFrame eor x3, x3, x3 cbz x2, L_scalarDone // Compare one byte at a time until s1 has vector alignment. 0: tst x0, #(kVectorSize-1) b.eq L_s1aligned ldrb w4, [x0],#1 // load byte from src1 ldrb w5, [x1],#1 // load byte from src2 subs x3, x4, x5 // if the are not equal ccmp w4, #0, #4, eq // or we find an EOS b.eq L_scalarDone // return the difference subs x2, x2, #1 // decrement length b.ne 0b // continue loop if non-zero // We found a mismatch or EOS before s1 became aligned. Simply return the // difference between the last bytes that we loaded. L_scalarDone: mov x0, x3 ClearFrameAndReturn L_s1aligned: // If s2 is similarly aligned to s1, then we can use a naive vector comparison // from this point on without worrying about spurious page faults; none of our // loads will ever cross a page boundary, because they are all aligned. #if HAS_MTE // Ensure that at least one comparison happened with MTE enabled, as we will // run with TCO (Tag Check Override) enabled for the majority of the comparison. ldrb w4, [x0] ldrb w5, [x1] #if DEVELOPMENT || DEBUG // We do not support TCO nesting, ensure that we got here with the expected // TCO state. bl EXT(mte_validate_tco_state) #endif /* DEVELOPMENT || DEBUG */ // We'll run the rest of the comparison with TCO enabled. msr TCO, #1 #endif /* HAS_MTE */ tst x1, #(kVectorSize-1) b.eq L_naiveVector /***************************************************************************** * Careful chunk comparison * *****************************************************************************/ // Otherwise, we need to be careful; although vector loads from s1 cannot // cross a page boundary because they are aligned, s2 is not aligned. We // compute the multiple of vector size that we can safely load before reaching // a page boundary, and compare only that far before switching over to scalar // comparisons to step across the page boundary. If this number happens to // be zero, we jump directly to the scalar comparison. neg x7, x1 ands x7, x7, #(PAGE_MIN_SIZE-kVectorSize) b.eq 2f .align 4 // If n is less than the number of bytes before a page-crossing load, jump // into the naive vector path instead, since we will not even reach a page // crossing. Otherwise, decrement n by that number before we monkey with it, // and set the decremented value aside. 0: cmp x2, x7 b.ls L_naiveVector sub x6, x2, x7 // Use vector comparisons until a mismatch or EOS is encountered, or the next // vector load from s2 would be page-crossing. 1: ldr q0, [x0],#(kVectorSize) ldr q1, [x1],#(kVectorSize) cmeq.16b v1, v0, v1 and.16b v0, v0, v1 // contains zero byte iff mismatch or EOS uminv.16b b1, v0 fmov w3, s1 // zero only iff comparison is finished cbz w3, L_vectorDone subs x7, x7, #(kVectorSize) b.ne 1b // Restore the updated n to x2 mov x2, x6 // The next vector load will cross a page boundary. Instead, compare one byte // at a time until s1 again has vector alignment, at which point we will have // compared exactly 16 bytes. 2: ldrb w4, [x0],#1 // load byte from src1 ldrb w5, [x1],#1 // load byte from src2 subs x3, x4, x5 // if the are not equal ccmp w4, #0, #4, eq // or we find an EOS #if HAS_MTE b.eq L_TCOscalarDone #else b.eq L_scalarDone // return the difference #endif subs x2, x2, #1 // decrement length #if HAS_MTE b.eq L_TCOscalarDone #else b.eq L_scalarDone // exit loop if zero. #endif tst x0, #(kVectorSize-1) b.ne 2b // Having compared one vector's worth of bytes using a scalar comparison, we // know that we are safely across the page boundary. Initialize x7 and jump // back into the vector comparison part of the loop. mov x7, #(PAGE_MIN_SIZE-kVectorSize) b 0b #if HAS_MTE L_TCOscalarDone: // Reset TCO state. msr TCO, #0 // Disable TCO, tag checking is enabled ldrb w4, [x0, #-1] ldrb w5, [x1, #-1] sub x0, x4, x5 ClearFrameAndReturn #endif /***************************************************************************** * Naive vector comparison * *****************************************************************************/ L_naiveVector: subs x3, x2, #(kVectorSize) b.lo L_scalar add x4, x0, x3 // save the addresses of the last vectors add x5, x1, x3 mov x2, x3 // length -= kVectorSize .align 4 0: ldr q0, [x0],#(kVectorSize) ldr q1, [x1],#(kVectorSize) cmeq.16b v1, v0, v1 and.16b v0, v0, v1 // contains zero byte iff mismatch or EOS uminv.16b b1, v0 fmov w3, s1 // zero only iff comparison is finished cbz w3, L_vectorDone subs x2, x2, #(kVectorSize) b.hi 0b // compare the last vector mov x0, x4 mov x1, x5 ldr q0, [x0],#(kVectorSize) ldr q1, [x1],#(kVectorSize) cmeq.16b v1, v0, v1 and.16b v0, v0, v1 // contains zero byte iff mismatch or EOS uminv.16b b1, v0 fmov w3, s1 // zero only iff comparison is finished cbz w3, L_vectorDone L_readNBytes: #if HAS_MTE msr TCO, #0 // Disable TCO, tag checking is enabled ldrb w4, [x0, #-1] ldrb w5, [x1, #-1] #endif eor x0, x0, x0 ClearFrameAndReturn L_vectorDone: #if HAS_MTE msr TCO, #0 // Disable TCO, tag checking is enabled #endif // Load the bytes corresponding to the first mismatch or EOS and return // their difference. eor.16b v1, v1, v1 cmhi.16b v0, v0, v1 // force non-zero lanes to 0xff ldr q1, L_mask orr.16b v0, v0, v1 // lane index in lanes containing mismatch or EOS uminv.16b b1, v0 fmov w3, s1 sub x3, x3, #(kVectorSize) ldrb w4, [x0, x3] ldrb w5, [x1, x3] sub x0, x4, x5 ClearFrameAndReturn L_scalar: #if HAS_MTE msr TCO, #0 // Disable TCO, tag checking is enabled #endif 0: ldrb w4, [x0],#1 // load byte from src1 ldrb w5, [x1],#1 // load byte from src2 subs x3, x4, x5 // if the are not equal ccmp w4, #0, #4, eq // or we find an EOS b.eq 1f // return the difference subs x2, x2, #1 // decrement length b.ne 0b // continue loop if non-zero 1: mov x0, x3 ClearFrameAndReturn |